Client Resources

Getting Started

Getting Started icon

Welcome to OSC! If you are new to supercomputing, new to OSC, or simply interested in getting an account (if you don't already have one), we have some resources to help you.

New User Resource Guide

Welcome to OSC!

This document provides some useful links to get you started.

How to request an account

For information on getting an account and allocation with us, please visit

https://www.osc.edu/supercomputing/support/account

OSC Client Portal

This is our web portal for managing your account, project, authorized users, etc. Documentation can be found here

https://www.osc.edu/supercomputing/portals/client_portal

Getting connected

Logging in to OSC resources is primarily accomplished via SSH (or “secure shell”). This is a method of getting a command-line interface on our clusters, which runs a version of the Linux operating system. For more details on how to connect to our systems, please visit

            http://www.osc.edu/supercomputing/getting-started/getting-connected

If that sounds too intimidating, and you would like to try a zero-client web-browser approach to using OSC, you can log in to our web portal, OnDemand:

            https://ondemand.osc.edu/

For an introduction to the features of OnDemand, check out

            https://www.osc.edu/resources/online_portals/ondemand

Transferring Files

SFTP (Secure File Transfer Protocol) is the primary method of moving files to and from OSC storage systems. SCP (Secure CoPy) is another common method. Small files can be moved to and from OSC systems inside of OnDemand. Moving large files to or from OSC via the login nodes can experience difficulty since there are compute restrictions on those nodes, so the recommendation is to connect to sftp.osc.edu or scp.osc.edu. If you have a large amount of data to move, you can schedule transfers via Globus Online.

Submitting Jobs

Instructions to be executed by the node(s) assigned to you when your request for resources can be met by the scheduler – can get very complicated, so it’s best to refer to the documentation for the software you want to use to get started.

            http://www.osc.edu/supercomputing/software/

We also have a more detailed guide on the batch system that we recommend new users read. It provides information about scheduling, job submission and how to write scripts to accomplish various tasks.

            http://www.osc.edu/supercomputing/batch-processing-at-osc

Account Maintenance and Usage

Please use MyOSC to update your password, shell, and check usage on the project. The command OSCusage will also provide usage details. For Ohio academics, you are charged according to these practices. Commercial clients are charged per core hour at their agreed-upon rate (1 RU is roughly equal to 10 core hours).

Training

To get an introduction to High-Performance Computing, please read our HPC Basics page.

            https://www.osc.edu/resources/getting_started/hpc_basics

If you want to learn more about using the command line, we have some useful resources on UNIX Basics.

            https://www.osc.edu/documentation/tutorials/unix-basics

For detailed instructions on how to perform some important tasks on our systems, check out our HOWTO articles.

            https://www.osc.edu/resources/getting_started/howto

Need Help?

Support can be found by contacting OSC Help weekdays, 9AM to 5PM (1-800-686-6472 or 614-292-1800). 
Fill out a request online.  You can also schedule a consultation (remote or in-person) with our staff. For remote consultations, please include "REMOTE" in the request.

We regularly maintain our webpages to show relevant events at the center (including training) and system notices on our main page (osc.edu). We also provide important information in the “message of the day” (visible when you log in). Furthermore, you can receive notices by following @HPCNotices on Twitter. 

More Useful Links:

FAQ: http://www.osc.edu/supercomputing/faq

Main Supercomputing pages: https://www.osc.edu/services/cluster_computing

New User Presentation Slides: https://www.osc.edu/~kcahill/NewUser

 

Documentation Attachment: 
Supercomputer: 

HPC Basics

Overview

HPC, or High Performance Computing, generally refers to aggregating computing resources together in order to perform more computing operations at once.

Basic definitions

  • Core (processor) - A single unit that executes a single chain of instructions.
  • Node - a single computer or server.
  • Cluster - many nodes connected together which are able to coordinate between themselves.

HPC Workflow

Using HPC is a little different from running programs on your desktop. When you login you’ll be connected to one of the system’s “login nodes”. These nodes serve as a staging area for you to marshal your data and submit jobs to the batch scheduler. Your job will then wait in a queue along with other researchers' jobs. Once the resources it requires become available, the batch scheduler will then run your job on a subset of our hundreds of “compute nodes”. You can see the overall structure in the diagram below.

 

Diagram: Several connected parts illustrating the layout of an OSC cluster. Users connect to one of a few "login nodes", which in turn connect to the "batch system", which runs jobs on a subset of the "compute nodes". The "shared filesystem" is connected to both the login nodes and the compute nodes.

 

HPC Citizenship

An important point about the diagram above is that OSC clusters are a collection of shared, finite resources. When you connect to the login nodes, you are sharing their resources (CPU cycles, memory, disk space, network bandwidth, etc.) with a few dozen other researchers. The same is true of the file servers when you access your home or project directories, and can even be true of the compute nodes.

For most day-to-day activities you should not have to worry about this, and we take precautions to limit the impact that others might have on your experience. That said, there are a few use cases that are worth watching out for:

  • The login nodes should only be used for light computation; any CPU- or memory-intensive operations should be done using the batch system. A good rule of thumb is that if you wouldn't want to run a task on your personal desktop because it would slow down other applications, you shouldn't run it on the login nodes. (See also: Interactive Jobs.)

  • I/O-intensive jobs should copy their files to fast, temporary storage, such as the local storage allocated to jobs or the Scratch parallel filesystem.

  • When running memory-intensive or potentially unstable jobs, we highly recommend requesting whole nodes.  By doing so you prevent other users jobs from being impacted by your job.

  • If you request partial nodes, be sure to consider the amount of memory available per core. (See: HPC Hardware.) If you need more memory, request more cores. It is perfectly acceptable to leave cores idle in this situation; memory is just as valuable a resource as processors.

In general, we just encourage our users to remember that what you do may affect other researchers on the system. If you think something you want to do or try might interfere with the work of others, we highly recommend that you contact us at oschelp@osc.edu.

Getting Connected

There are two ways to connect to our systems. The traditional way will require you to install some software locally on your machine, including an SSH client, SFTP client, and optionally an X Windows server. The alternative is to use our zero-client web portal, OnDemand.

OnDemand Web Portal

OnDemand is our "one stop shop" for access to our High Performance Computing resources. With OnDemand, you can upload and download files, create, edit, submit, and monitor jobs, run GUI applications, and connect via SSH, all via a web broswer, with no client software to install and configure.

You can access OnDemand by pointing a web browser to ondemand.osc.edu. Documentation is available here. Any newer version of a common web brower should be sufficient to connect.

Using Traditional Clients

Required Software

In order to use our systems, you'll need two main pieces of software: an SFTP client and an SSH client.

SFTP ("SSH File Transfer Protocol") clients allow you transfer files between your workstation and our shared filesystem in a secure manner. We recommend the following applications:

  • FileZilla: A high-performance open-source client for Windows, Linux, and OS X. A guide to using FileZilla is available here (external).
  • CyberDuck: A high quality free client for Windows and OS X.
  • sftp: The command-line utility sftp comes pre-installed on OS X and most Linux systems.

SSH ("Secure Shell") clients allow you to open a command-line-based "terminal session" with our clusters. We recommend the following options:

  • PuTTY: A simple, open-source client for Windows.
  • Secure Shell for Google Chrome: A free, HTML5-based SSH client for Google Chrome.
  • ssh: The command-line utility ssh comes pre-installed on OS X and most Linux systems.

A third, optional piece of software you might want to install is an X Windows server, which will be necessary if you want to run graphical, windowed applications like MATLAB. We recommend the following X Windows servers:

  • Xming: Xming offers a free version of their X Windows server for Microsoft Windows systems.
  • X-Win32: StarNet's X-Win32 is a commercial X Windows server for Microsoft Windows systems. They offer a free, thirty-day trial.
  • X11.app/XQuartz: X11.app, an Apple-supported version of the open-source XQuartz project, is freely available for OS X. 

In addition, for Windows users, you can use OSC Connectwhich is a native windows application developed by OSC to provide a launcher for secure file transfer, VNC, terminal, and web based services, as well as preconfigured management of secure tunnel connections. See this page for more information on OSC Connect. 

Connecting via SSH

The primary way you'll interact with the OSC clusters is through the SSH terminal. See our supercomputing environments for the hostnames of our current clusters. You should not need to do anything special beyond entering the hostname.

Once you've established an SSH connection, you will be presented with some informational text about the cluster you've connected to followed by a UNIX command prompt. For a brief discussion of UNIX command prompts and what you can do with them, see the next section of this guide.

Transferring Files

To transfer files, use your preferred SFTP client to connect to:

sftp.osc.edu

You may see warning message including SSH key fingerprint. Verify that the fingerprint in the message matches one of the SSH key fingerprint listed here, then type yes.

Since process times are limited on the login nodes, trying to transfer large files directly to pitzer.osc.edu or other login nodes may terminate partway through. The sftp.osc.edu is specially configured to avoid this issue, and so we recommend it for all your file transfers.

Note: The sftp.osc.edu host is not connected to the scheduler, so you cannot submit jobs from this host. Use of this host for any purpose other than file transfer is not permitted.

Firewall Configuration 

See our Firewall and Proxy Settings page for information on how to configure your firewall to allow connection to and from OSC.

Setting up X Windows (Optional)

With an X Windows server you will be able to run graphical applications on our clusters that display on your workstation. To do this, you will need to launch your X Windows server before connecting to our systems. Then, when setting up your SSH connection, you will need to be sure to enable "X11 Forwarding".

For users of the command-line ssh client, you can do this by adding the "-X" option. For example, the below will connect to the Pitzer cluster with X11 forwarding:

$ ssh -X username@pitzer.osc.edu

If you are connecting with PuTTY, the checkbox to enable X11 forwarding can be found in the connections pane under "Connections → SSH → X11".

For other SSH clients, consult their documentation to determine how to enable X11 forwarding.

NOTE: The X-Windows protocol is not a high-performance one. Depending on your system and Internet connection, X Windows applications may run very slowly, even to the point of being unusable. If you find this to be the case and graphical applications are a necessity for your work, please contact OSC Help to discuss alternatives.
Supercomputer: 
Service: 

Budgets and Accounts

The Ohio Supercomputer Center provides services to clients from a variety of types of organizations. The methods for gaining access to the systems are different between Ohio academic institutions and everyone else.

Ohio academic clients

Primarily, our users are Ohio-based and academic, and the vast majority of our resources will continue to be consumed by Ohio-based academic users. See our fee structure FAQ.

Other clients

Other users (business, non-Ohio academic, nonprofit, hospital, etc.) interested in using Center resources may purchase services at a set rate available on our price list. Expert consulting support is also available.

Other computing centers

For users interested in gaining access to larger resources, please contact OSC Help. We can assist you in applying for resources at an NSF or XSEDE site.

Managing an OSC project

Once a project has been created, the PI can create accounts for users by adding them through the client portal. Existing users can also be added. More information can be found on the Project Menu documentation page.

I need additional resources for my existing project and/or I received an email my allocation is exhausted

If an academic PI wants a new project or to update the budget balance on an existing project(s), please see our creating projects and budget documentation

I wish to use OSC to support teaching a class

We provide special classroom projects for this purpose and at no cost. You may use the client portal after creating an account. The request will need to include a syllabus or a similar document. 

I don't think I fit in the above categories

Please contact us in order to discuss options for using OSC resources.

Project Applications

Application if you are employed by an Ohio Academic Institution

Please contact OSC Help with questions and see the fee structure FAQ

Use of computing resources and services at OSC is subject to the Ohio Supercomputer Center (OSC) Code of Ethics for Academic Users.  Ohio Academic Clients are eligible for highly subsidized access to OSC resources, with fees only accruing after a credit provided is exhausted. Clients from an Ohio academic institution that expect to use more than the credit should consult with their institution on the proper guidance for requesting approval to be charged for usage. See the academic fee structure FAQ page for more information.

Eligible principal investigators (PIs) at Ohio academic institutions are able to request projects at OSC, but should also consult with their institution before incurring charges. In order to be an eligible PI at OSC, you must be eligible to hold PI status at your college, university, or research organization administered by an Ohio academic institution (i.e., be a full-time, permanent academic researcher or tenure-track faculty member at an Ohio college or university). Students, post-doctoral fellows, visiting scientists, and others who wish to use the facilities may be authorized users on projects headed by an eligible PI. Once a PI has received their project information, he/she can manage users for the project. Principal Investigators of OSC projects are responsible for updating their authorized user list, their outside funding sources, and their publications and presentations that cite OSC. All of these tasks can be accomplished by contacting using the client portal. Please review the documentation for more information. PIs are also responsible for monitoring their project's budget (balance) and for requesting a new budget (balance) before going negative, as projects with negative balances are restricted.

OSC's online project requests through our Client Portal is part of an electronic system that leads you through the process step by step. Before you begin to fill in the application form, especially if you are new to the process, look at the academic fee structure page.  You can save a partially completed project request for later use.

If you need assistance, please contact OSC Help.

Application if you are NOT employed by an Ohio Academic Institution

Researchers from businesses, non-Ohio academic, nonprofits, hospitals or other organizations (which do not need to be based in Ohio) who wish to use the OSC's resources should complete the Other Client request form available here. All clients not affiliated with and approved by an Ohio academic institution must sign a service agreement, provide a $500 deposit, and pay for resource usage per a standard price list.

Letter of Commitment for Outside Funding Proposals

OSC will provide a letter of commitment users can include with their account proposals for outside funding, such as from the Department of Energy, National Institutes of Health, National Science Foundation (limited to standard text, per NSF policy), etc. This letter details OSC's commitment to supporting research efforts of its users and the facilities and platforms we provide our users. [Note: This letter does not waive the normal OSC budget process; it merely states that OSC is willing to support such research.] The information users must provide for the letter is:

  • address to which the letter should be directed (e.g. NSF, Department, mailing address)
  • name of funding agency's representative
  • name of the proposal, as well as names and home institutions of the Co-PIs
  • budget amount per year you would apply for if you were to receive funding
  • number of years of proposed research
  • solicitation number

Send e-mail with your request for the commitment letter to OSC Help or submit online. We will prepare a draft for your approval and then we will send you the final PDF for your proposal submission. Please allow at least two working days for this service.

Letter of Support for Outside Funding Proposals

Letters of support may be subject to strict and specific guidelines, and may not be accepted by your funding agency.

If you need a letter of support, please see above "Letter of Commitment for Outside Funding Proposals". 

Applying at NSF Centers

Researchers requiring additional computing resources should consider applying for allocations at National Science Foundation Centers. For more information, please write to oschelp@osc.edu, and your inquiry will be directed to the appropriate staff member.:

We require you cite OSC in any publications or reports that result from projects supported by our services.

UNIX Basics

OSC HPC resources use an operating system called "Linux", which is a UNIX-based operating system, first released on 5 October 1991. Linux is by a wide margin the most popular operating system choice for supercomputing, with over 90% of the Top 500 list running some variant of it. In fact, many common devices run Linux variant operating systems, including game consoles, tablets, routers, and even Android-based smartphones.

While Linux supports desktop graphical user interface configurations (as does OSC) in most cases, file manipulation will be done via the command line. Since all jobs run in batch will be non-interactive, they by definition will not allow the use of GUIs. Thus, we strongly suggest new users become comfortable with basic command-line operations, so that they can learn to write scripts to submit to the scheduler that will behave as intended. We have provided some tutorials explaining basics from moving about the file system, to extracting archives, to modifying your environment, that are available for self-paced learning.

Linux Command Line Fundamentals

This tutorial teaches you about the linux command line and shows you some useful commands. It also shows you how to get help in linux by using the man and apropos commands.


Linux Tutorial

This tutorial guides you through the process of creating and submitting a batch script on one of our compute clusters. This is a linux tutorial which uses batch scripting as an example, not a tutorial on writing batch scripts. The primary goal is not to teach you about batch scripting, but for you to become familiar with certain linux commands that can be used either in a batch script or at the command line. There are other pages on the OSC web site that go into the details of submitting a job with a batch script.


Linux Shortcuts

This tutorial shows you some handy time-saving shortcuts in linux. Once you have a good understanding of how the command line works, you will want to learn how to work more efficiently.


Tar Tutorial

This tutorial shows you how to download tar (tape archive) files from the internet and how to deal with large directory trees of files.


Service: 

Linux Command Line Fundamentals

 

Description

This tutorial teaches you about the linux command line and shows you some useful commands. It also shows you how to get help in linux by using the man and apropos commands.

For more training and practice using the command line, you can find many great tutorials. Here are a few:

https://www.learnenough.com/command-line-tutorial

https://cvw.cac.cornell.edu/Linux/

http://www.ee.surrey.ac.uk/Teaching/Unix/

https://www.udacity.com/course/linux-command-line-basics--ud595

 

More Advanced:

http://moo.nac.uci.edu/~hjm/How_Programs_Work_On_Linux.html

Prerequisites

None.

Introduction

Unix is an operating system that comes with several application programs. Other examples of operating systems are Microsoft Windows, Apple OS and Google's Android. An operating system is the program running on a computer (or a smartphone) that allows the user to interact with the machine -- to manage files and folders, perform queries and launch applications. In graphical operating systems, like Windows, you interact with the machine mainly with the mouse. You click on icons or make selections from the menus. The Unix that runs on OSC clusters gives you a command line interface. That is, the way you tell the operating system what you want to do is by typing a command at the prompt and hitting return. To create a new folder you type mkdir. To copy a file from one folder to another, you type cp. And to launch an application program, say the editor emacs, you type the name of the application. While this may seem old-fashioned, you will find that once you master some simple concepts and commands you are able to do what you need to do efficiently and that you have enough flexibility to customize the processes that you use on OSC clusters to suit your needs.

Common Tasks on OSC Clusters

What are some common tasks you will perform on OSC clusters? Probably the most common scenario is that you want to run some of the software we have installed on our clusters. You may have your own input files that will be processed by an application program. The application may generate output files which you need to organize. You will probably have to create a job script so that you can execute the application in batch mode. To perform these tasks, you need to develop a few different skills. Another possibility is that you are not just a user of the software installed on our clusters but a developer of your own software -- or maybe you are making some modifications to an application program so you need to be able to build the modified version and run it. In this scenario you need many of the same skills plus some others. This tutorial shows you the basics of working with the Unix command line. Other tutorials go into more depth to help you learn more advanced skills.

The Kernel and the Shell

You can think of Unix as consisting of two parts -- the kernel and the shell. The kernel is the guts of the Unix operating system -- the core software running on a machine that performs the infrastructure tasks like making sure multiple users can work at the same time. You don't need to know anything about the kernel for the purposes of this tutorial. The shell is the program that interprets the commands you enter at the command prompt. There are several different flavors of Unix shells -- Bourne, Korn, Cshell, TCshell and Bash. There are some differences in how you do things in the different shells, but they are not major and they shouldn't show up in this tutorial. However, in the interest of simplicity, this tutorial will assume you are using the Bash shell. This is the default shell for OSC users. Unless you do something to change that, you will be running the Bash shell when you log onto Owens or Pitzer.

The Command Prompt

The first thing you need to do is log onto one of the OSC clusters, Owens or Pitzer. If you do not know how to do this, you can find help at the OSC home page. If you are connecting from a Windows system, you need to download and setup the OSC Starter Kit which you can find here. If you are connecting from a Mac or Linux system, you will use ssh. To get more information about using ssh, go to the OSC home page, hold your cursor over the "Supercomputing" menu in the main blue menu bar and select "FAQ." This should help you get started. Once you are logged in look for the last thing displayed in the terminal window. It should be something like

-bash-3.2$

with a block cursor after it. This is the command prompt -- it's where you will see the commands you type in echoed to the screen. In this tutorial, we will abbreviate the command prompt with just the dollar sign - $. The first thing you will want to know is how to log off. You can log off of the cluster by typing "exit" then typing the <Enter> key at the command prompt:

$ exit <Enter>

For the rest of this tutorial, when commands are shown, the <Enter> will be omitted, but you must always enter <Enter> to tell the shell to execute the command you just typed.

First Simple Commands

So let's try typing a few commands at the prompt (remember to type the <Enter> key after the command):

$ date

$ cal

$ finger

$ who

$ whoami

$ finger -l

That last command is finger followed by a space then a minus sign then the lower case L. Is it obvious what these commands do? Shortly you will learn how to get information about what each command does and how you can make it behave in different ways. You should notice the difference between "finger" and "finger -l" -- these two commands seem to do similar things (they give information about the users who are logged in to the system) but they print the information in different formats. try the two commands again and examine the output. Note that you can use the scroll bar on your terminal window to look at text that has scrolled off the screen.

man

The "man" command is how you find out information about what a command does. Type the following command:

$ man

It's kind of a smart-alecky answer you get back, but at least you learn that "man" is short for "manual" and that the purpose is to print the manual page for a command. Before we start looking at manual pages, you need to know something about the way Unix displays them. It does not just print the manual page and return you to the command prompt -- it puts you into a mode where you are interactively viewing the manual page. At the bottom of the page you should see a colon (:) instead of the usual command prompt (-bash-3.2$). You can move around in the man page by typing things at the colon. To exit the man page, you need to type a "q" followed by <Enter>. So try that first. Type

$ man finger

then at the colon of the man page type

: q

You do not have to type <Enter> after the "q" (this is different from the shell prompt.) You should be back at the shell prompt now. Now let's go through the man page a bit. Once again, type

$ man finger

Now instead of just quitting, let's look at the contents of the man page. The entire man page is probably not displayed in your terminal. To scroll up or down, use the arrow keys or the <Page Up> and <Page Down> keys of the keyboard. The <Enter> and <Space> keys also scroll. Remember that "q" will quit out of the man page and get you back to the shell prompt.

The first thing you see is a section with the heading "NAME" which displays the name of the command and a short summary of what it does. Then there is a section called "SYNOPSIS" which shows the syntax of the command. In this case you should see

SYNOPSIS

     finger [-lmsp] [user ...] [user@host ...]

Remember how "finger" and "finger -l" gave different output? The [-lmsp] tells you that you can use one of those four letters as a command option -- i.e., a way of modifying the way the command works. In the "DESCRIPTION" section of the man page you will see a longer description of the command and an explanation of the options. Anything shown in the command synopsis which is contained within square brackets ([ ]) is optional. That's why it is ok to type "finger" with no options and no user. What about "user" -- what is that? To see what that means, quit out of the man page and type the following at the command prompt:

$ whoami

Let's say your username is osu0000. Then the result of the "whoami" command is osu0000. Now enter the following command (but replace osu0000 with your username):

$ finger osu0000

You should get information about yourself and no other users. You can also enter any of the usernames that are output when you enter the "finger" command by itself. The user names are in the leftmost column of output. Now try

 

$ finger -l osu0000

$ finger -lp osu0000

$ finger -s osu0000 osu0001

For the last command, use your username and the username of some other username that shows up in the output of the "finger" command with no arguments.

Note that a unix command consists of three parts:

  • command
  • option(s)
  • argument(s)

You don't necessarily have to enter an argument (as you saw with the "finger" command) but sometimes a command makes no sense without an argument so you must enter one -- you saw this with the "man" command. Try typing

$ man man

and looking briefly at the output. One thing to notice is the synopsis -- there are a lot of possible options for the "man" command, but the last thing shown in the command synopsis is "name ..." -- notice that "name" is not contained in square brackets. This is because it is not optional -- you must enter at least one name. What happens if you enter two names?

$ man man finger

The first thing that happens is you get the man page for the "man" command. What happens when you quit out of the man page? You should now get the man page for the "finger" command. If you quit out of this one you will be back at the shell prompt.

Combining Commands

You can "pipe" the output of one command to another. First, let's learn about the "more" command:

$ man more

Read the "DESCRIPTION" section -- it says that more is used to page through text that doesn't fit on one screen. It also recommends that the "less" command is more powerful. Ok, so let's learn about the "less" command:

$ man less

You see from the description that "less" also allows you to examine text one screenful at a time. Does this sound familiar? The "man" command actually uses the "less" command to display its output. But you can use the "less" command yourself. If you have a long text file named "foo.txt" you could type

$ less foo.txt

and you would be able to examine the contents of the file one screen at a time. But you can also use "less" to help you look at the output of a command that prints more than one screenful of output. Try this:

$ finger | less

That's "finger" followed by a space followed by the vertical bar (shifted backslash on most keyboards) followed by a space followed by "less" followed by <Enter>. You should now be looking at the output of the "finger" command in an interactive fashion, just as you were looking at man pages. Remember, to scroll use the arrow keys, the <Page Up> and <Page Down> keys, the <Enter> key or the space bar; and to quit, type "q".

Now try the following (but remember to replace "osu0000" with your actual username):

$ finger | grep osu0000

The "grep" command is Unix's command for searching. Here you are telling Unix to search the output of the "finger" command for the text "osu0000" (or whatever your username is.)

If you try to pipe the output of one command to a second command and the second is a command which works with no arguments, you won't get what you expect. Try

$ whoami | finger

You see that it does not give the same output as

$ finger osu0000

(assuming "whoami" returns osu0000.)

In this case what you can do is the following:

$ finger `whoami`

That's "finger" space backquote "whoami" backquote. The backquote key is to the left of the number 1 key on a standard keyboard.

apropos

Enter the following command:

$ man apropos

As you can see, the apropos searches descriptions of commands and finds commands whose descriptions match the keyword you entered as the argument. That means it outputs a list of commands that have something to do with the keyword you entered. Try this

$ apropos

Ok, you need to enter an argument for the "apropos" command.

So try

$ apropos calendar

Now you see that among the results are two commands -- "cal" and "difftime" that have something to do with the keyword "calendar."

Linux Tutorial

 

Description

This tutorial guides you through the process of creating and submitting a batch script on one of our compute clusters. This is a linux tutorial which uses batch scripting as an example, not a tutorial on writing batch scripts. The primary goal is not to teach you about batch scripting, but for you to become familiar with certain linux commands. There are other pages on the OSC web site that go into the details of submitting a job with a batch script.

Prerequisites

Goals

  • Create subdirectories to organize information
  • Create a batch script with a text editor
  • Submit a job
  • Check on the progress of the job
  • Change the permissions of the output files
  • Get familiar with some common unix commands

Step 1 - Organize your directories

When you first log in to our clusters, you are in your home directory. For the purposes of this illustration, we will pretend you are user osu0001 and your project code is PRJ0001, but when you try out commands you must use your own username and project code.

$ pwd
/users/PRJ0001/osu0001
 
Note: you will see your user name and a different number after the /users.
 
It's a good idea to organize your work into separate directories. If you have used Windows or the Mac operating system, you may think of these as folders. Each folder may contain files and subfolders. The subfolders may contain other files and subfolders of their own. In linux we use the term "directory" instead of "folder." Use directories to organize your work.
 
Type the following four lines and take note of the output after each one:
 
$ touch foo1
$ touch foo2
$ ls
$ ls -l
$ ls -lt
$ ls -ltr
 
The "touch" command just creates an empty file with the name you give it.
You probably already know that the ls command shows the contents of the current working directory; that is, the directory you see when you type pwd. But what is the point of the "-l", "-lt" or "-ltr"? You noticed the difference in the output between just the "ls" command and the "ls -l" command.
Most unix commands have options you can specify that change the way the command works. The options can be specified by the "-" (minus sign) followed by a single letter. "ls -ltr" is actually specifying three options to the ls command.
l: I want to see the output in long format -- one file per line with some interesting information about each file
t: sort the display of files by when they were last modified, most-recently modified first
r: reverse the order of display (combined with -t this displays the most-recently modified file last -- it should be BatchTutorial in this case.)
 
I like using "ls -ltr" because I find it convenient to see the most recently modified file at the end of the list.
 
Now try this:
$ mkdir BatchTutorial
$ ls -ltr
 
The "mkdir" command makes a new directory with the name you give it. This is a subfolder of the current working directory. The current working directory is where your current focus is in the hierarchy of directories. The 'pwd' command shows you are in your home directory:
 
$ pwd
/users/PRJ0001/osu0001
 
Now try this:
$ cd BatchTutorial
$ pwd
 
What is the output of 'pwd' now? "cd" is short for "change directory" -- think of it as moving you into a different place in the hierarchy of directories. Now do
$ cd ..
$ pwd
Where are you now?

Step 2 -- Get familiar with some more unix commands

Try the following:

$ echo where am I?
$ echo I am in `pwd`
$ echo my home directory is $HOME
$ echo HOME
$ echo this directory contains `ls -l`

These examples show what the echo command does and how to do some interesting things with it. The `pwd` means the result of issuing the command pwd. HOME is an example of an environment variable. These are strings that stand for other strings. HOME is defined when you log in to a unix system. $HOME means the string the variable HOME stands for. Notice that the result of "echo HOME" does not do the substitution. Also notice that the last example shows things don't always get formatted the way you would like.

Some more commands to try:

$ cal
$ cal > foo3
$ cat foo3
$ whoami
$ date

Using the ">" after a command puts the output of the command into a file with the name you specify. The "cat" command prints the contents of a file to the screen.

Two very important UNIX commands are the cp and mv commands. Assume you have a file called foo3 in your current directory created by the "cal > foo3" command. Suppose you want to make a copy of foo3 called foo4. You would do this with the following command:

$ cp foo3 foo4
$ ls -ltr

Now suppose you want to rename the file 'foo4' to 'foo5'. You do this with:

$ mv foo4 foo5
$ ls -ltr

'mv' is short for 'move' and it is used for renaming files. It can also be used to move a file to a different directory.

$ mkdir CalDir
$ mv foo5 CalDir
$ ls
$ ls CalDir

Notice that if you give a directory with the "ls" command is shows you what is in that directory rather than the current working directory.

Now try the following:

$ ls CalDir
$ cd CalDir
$ ls
$ cd ..
$ cp foo3 CalDir
$ ls CalDir

Notice that you can use the "cp" command to copy a file to a different directory -- the copy will have the same name as the original file. What if you forget to do the mkdir first?

$ cp foo3 FooDir

Now what happens when you do the following:

$ ls FooDir
$ cd FooDir
$ cat CalDir
$ cat FooDir
$ ls -ltr

CalDir is a directory, but FooDir is a regular file. You can tell this by the "d" that shows up in the string of letters when you do the "ls -ltr". That's what happens when you try to cp or mv a file to a directory that doesn't exist -- a file gets created with the target name. You can imagine a scenario in which you run a program and want to copy the resulting files to a directory called Output but you forget to create the directory first -- this is a fairly common mistake.

Step 3 -- Environment Variables

Before we move on to creating a batch script, you need to know more about environment variables. An environment variable is a word that stands for some other text. We have already seen an example of this with the variable HOME. Try this:

$ MY_ENV_VAR="something I would rather not type over and over"
$ echo MY_ENV_VAR
$ echo $MY_ENV_VAR
$ echo "MY_ENV_VAR stands for $MY_ENV_VAR"

You define an environment variable by assigning some text to it with the equals sign. That's what the first line above does. When you use '$' followed by the name of your environment variable in a command line, UNIX makes the substitution. If you forget the '$' the substitution will not be made.

There are some environment variables that come pre-defined when you log in. Try using 'echo' to see the values of the following variables: HOME, HOSTNAME, SHELL, TERM, PATH.

Now you are ready to use some of this unix knowledge to create and run a script.

Step 4 -- Create and run a script

Before we create a batch script and submit it to a compute node, we will do something a bit simpler. We will create a regular script file that will be run on the login node. A script is just a file that consists of unix commands that will run when you execute the script file. It is a way of gathering together a bunch of commands that you want to execute all at once. You can do some very powerful things with scripting to automate tasks that are tedious to do by hand, but we are just going to create a script that contains a few commands we could easily type in. This is to help you understand what is happening when you submit a batch script to run on a compute node.

Use a text editor to create a file named "tutorial.sh" which contains the following text (note that with emacs or nano you can use the mouse to select text and then paste it into the editor with the middle mouse button):

$ nano tutorial.sh

 

echo ----
echo Job started at `date`
echo ----
echo This job is working on node `hostname`

SH_WORKDIR=`pwd`
echo working directory is $SH_WORKDIR
echo ----
echo The contents of $SH_WORKDIR
ls -ltr
echo
echo ----
echo
echo creating a file in SH_WORKDIR
whoami > whoami-sh-workdir

SH_TMPDIR=${SH_WORKDIR}/sh-temp
mkdir $SH_TMPDIR
cd $SH_TMPDIR
echo ----
echo TMPDIR IS `pwd`
echo ----
echo wait for 12 seconds
sleep 12
echo ----
echo creating a file in SH_TMPDIR
whoami > whoami-sh-tmpdir

# copy the file back to the output subdirectory
cp ${SH_TMPDIR}/whoami-sh-tmpdir ${SH_WORKDIR}/output

cd $SH_WORKDIR

echo ----
echo Job ended at `date`

To run it:

$ chmod u+x tutorial.sh
$ ./tutorial.sh

Look at the output created on the screen and the changes in your directory to see what the script did.

Step 5 -- Create and run a batch job

Use your favorite text editor to create a file called tutorial.pbs in the BatchTutorial directory which has the following contents (remember, you can use the mouse to cut and paste text):

#PBS -l walltime=00:02:00
#PBS -l nodes=1:ppn=1
#PBS -N foobar
#PBS -j oe
#PBS -r n

echo ----
echo Job started at `date`
echo ----
echo This job is working on compute node `cat $PBS_NODEFILE`

cd $PBS_O_WORKDIR
echo show what PBS_O_WORKDIR is
echo PBS_O_WORKDIR IS `pwd`
echo ----
echo The contents of PBS_O_WORKDIR:
ls -ltr
echo
echo ----
echo
echo creating a file in PBS_O_WORKDIR
whoami > whoami-pbs-o-workdir

cd $TMPDIR
echo ----
echo TMPDIR IS `pwd`
echo ----
echo wait for 42 seconds
sleep 42
echo ----
echo creating a file in TMPDIR
whoami > whoami-tmpdir

# copy the file back to the output subdirectory
pbsdcp -g $TMPDIR/whoami-tmpdir $PBS_O_WORKDIR/output

echo ----
echo Job ended at `date`
 
To submit the batch script, type
$ qsub tutorial.pbs
Use qstat -u [username] to check on the progress of your job. If you see something like this
$ qstat -u osu0001

                                                                             Req'd  Req'd   Elap
Job ID             Username    Queue    Jobname          SessID NDS   TSK    Memory Time  S Time
------------------ ----------- -------- ---------------- ------ ----- ------ ------ ----- - -----
458842.oak-batch   osu0001     serial   foobar              --      1      1    --  00:02 Q   --
 
this means the job is in the queue -- it hasn't started yet. That is what the "Q" under the S column means.
 
If you see something like this:
                                                                             Req'd  Req'd   Elap
Job ID             Username    Queue    Jobname          SessID NDS   TSK    Memory Time  S Time
------------------ ----------- -------- ---------------- ------ ----- ------ ------ ----- - -----
458842.oak-batch   osu0001     serial   foobar            26276     1      1    --  00:02 R   --
this means the job is running and has job id 458842.
 
When the output of the qstat command is empty, the job is done.
 
After it is done, there should be a file called "foobar.o458842" in the directory.
Note that your file will end with a different number -- namely the job id number assigned to your job.
Check this with
$ ls -ltr
$ cat foobar.oNNNNNN

Where (NNNNNN is your job id).

The name of this file is determined by two things:
  1. The name you give the job in the script file with the header line #PBS -N foobar
  2. The job id number assigned to the job.

The name of the script file (tutorial.pbs) has nothing to do with the name of the output file.

Examine the contents of the output file foobar.oNNNNNN carefully. You should be able to see the results of some of the commands you put in tutorial.pbs. It also shows you the values of the variables PBS_NODEFILE, PBS_O_WORKDIR and TMPDIR. These variables exist only while your job is running. Try

$ echo $PBS_O_WORKDIR

and you will see it is no longer defined. $PBS_NODEFILE is a file which contains a list of all the nodes your job is running on. Because this script has the line

#PBS -l nodes=1:ppn=1

the contents of $PBS_NODEFILE is the name of a single compute node.

Notice that $TMPDIR is /tmp/pbstmp.NNNNNN (again, NNNNNN is the id number for this job.) Try

$ ls /tmp/pbstmp.NNNNNN

Why doesn't this directory exist? Because it is a directory on the compute node, not on the login node. Each machine in the cluster has its own /tmp directory and they do not contain the same files and subdirectories. The /users directories are shared by all the nodes (login or compute) but each node has its own /tmp directory (as well as other unshared directories.)

Tar Tutorial

 

Prerequisites

Step 1 -- Create a directory to work with and download a "tarball"

Start off with the following:

$ mkdir TarTutorial
$ cd TarTutorial
$ wget http://www.mmm.ucar.edu/wrf/src/WRFDAV3.1.tar.gz
$ ls -ltr

The third command will take a while because it is downloading a file from the internet. The file is call a "tarball" or a "gzipped tarball". TAR is an old unix short name for "tape archive" but a tar file is a file that contains a bunch of other files. If you have to move a bunch of files from one place to another, a good way to do it is to pack them into a tar file, move the tar file where you want it then unpack the files at the destination. A tar file usually has the extension ".tar". What about the ".gz"? This means the tar file has been further compressed with the program gzip -- this makes it a lot smaller.

Step 2 -- Unpack the "tarball" and check out the contents

After step 1 your working directory should be ~/TarTutorial and there should be a file called WRFDAV3.1.tar.gz in it.

Now do this:

$ gunzip WRFDAV3.1.tar.gz
$ ls -ltr

You should now have a file called WRFDAV3.1.tar which should be quite a bit larger in size than WRFDAV3.1.tar.gz -- this is because it has been uncompressed by the "gunzip" command which is the opposite of the "gzip" command.

Now do this:

$ tar -xvf WRFDAV3.1.tar
$ ls -ltr

You should see a lot of filenames go by on the screen and when the first command is done and you issue the ls command you should see two things -- WRFDAV3.1.tar is still there but there is also a directory called WRFDA. You can look at the contents of this directory and navigate around in the directory tree to see what is in there. The options on the "tar" command have the following meanings (you can do a "man tar" to get all the options):

x: extract the contents of the tar file

v: be verbose, i.e. show what is happening on the screen

f: the name of the file which follows the "f" option is the tar file to expand.

Another thing you can do is see how much space is being taken up by the files. Make sure TarTutorial is your working directory then issue the following command:

$ du .

Remember that "." (dot) means the current working directory. The "du" command means "disk usage" -- it shows you how much space is being used by every file and directory in the directory tree. It ends up with the highest level files and directories. You might prefer to do

$ du -h .
$ ls -ltrh

Adding the "-h" option to these commands puts the file sizes in human-readable format -- you should get a size of 66M for the tar file -- that's 66 megabytes -- and "du" should print a size of 77M next to ./WRFDA.

Step 3 -- create your own "tarball"

Now, make your own tar file from the WRFDA directory tree:

$ tar -cf mywrf.tar WRFDA
$ ls -ltrh

You have created a tar from all the files in the WRFDA directory. The options given to the "tar" command have the following meanings:

c: create a tar file

f: give it the name which follows the "f" option

The files WRFDAV3.1.tar and mywrf.tar are identical. Now compress the tar file you made:

$ gzip mywrf.tar
$ ls -ltrh

You should see a file called mywrf.tar.gz which is smaller than WRFDAV3.1.tar.

Step 4 -- Clean up!

You don't want to leave all these files lying around. So delete them

$ rm WRFDAV3.1.tar
$ rm mywrf.tar
$ rm WRFDA

Oops! You can't remove the directory. You need to use the "rmdir" command:

$ rmdir WRFDA

Oh no! That doesn't work on a directory that's not empty. So are you stuck with all those files? Maybe you can do this:

$ cd WRFDA
$ rm *
$ cd ..
$ rmdir WRFDA

That won't work either because there are some subdirectories in WRFDA and "rm *" won't remove them. Do you have to work your way to the all the leaves at the bottom of the directory tree and remove files then come back up and remove directories? No, there is a simpler way:

$ rm -Rf WRFDA

This will get rid of the entire directory tree. The options have the following meanings:

R: recursively remove all files and directories

f: force; i.e., just remove everything without asking for confirmation

I encourage you to do

$ man rm

and check out all the options. Or some of them -- there are quite a few.

 

Unix Shortcuts

 

Description

This tutorial shows you some handy time-saving shortcuts in linux. Once you have a good understanding of how the command line works, you will want to learn how to work more efficiently.

Prerequisites

Linux command line fundamentals.

Goals

  • Save you time when working on a linux system
  • Increase your appreciation of the power of linux

Step 1 -- The Arrow Keys

Note: even if you know how to use the up arrow in linux, you need to enter the commands in this section because they are used in the following sections. So to begin this tutorial, go to your home directory and create a new directory called ShortCuts:

$ cd
$ mkdir Shortcuts
$ cd Shortcuts

(If a directory or file named "Shortcuts" already exists, name it something else.)

Imagine typing in a long linux command and making a typo. This is one of the frustrating things about a command line interface -- you have to retype the command, correcting the typo this time. Or what if you have to type several similar commands -- wouldn't it be nice to have a way to recall a previous command, make a few changes, and enter the new command? This is what the up arrow is for.

Try the following:

$ cd ..
$ cd ShortCuts (type a capital C)

Linux should tell you there is no directory with that name.

Now type the up arrow key -- the previous command you entered shows up on the command line, and you can use the left arrow to move the cursor just after the capital C, hit Backspace, and type a lower case c. Note you can also position the cursor before the capital C and hit Delete to get rid of it.

Once you have changed the capital C to a lower case c you can hit Return to enter the command -- you do not have to move the cursor to the end of the line.

Now hit the up arrow key a few times, then hit the down arrow key and notice what happens. Play around with this until you get a good feel for what is happening.

Linux maintains a history of commands you have entered. Using the up and down arrow keys, you can recall previously-entered commands to the command line, edit them and re-issue them.

Note that in addition to the left and right arrow keys you can use the Home and End keys to move to the beginning or end of the command line. Also, if you hold down the Ctrl key when you type an arrow key, the cursor will move by an entire word instead of a single character -- this is useful is many situations and works in many editors.

Let's use this to create a directory hierarchy and a few files. Start in the Shortcuts directory and enter the following commands, using the arrow keys to simplify your job:

$ mkdir directory1
$ mkdir directory1/directory2
$ mkdir directory1/directory2/directory3
$ cd directory1/directory2/diectoryr3  (remember the Home key and the Ctrl key with left and right arrows)
$ hostname > file1
$ whoami > file2
$ mkdir directory4
$ cal > directory4/file3

Step 2 -- Using the TAB key

Linux has short, cryptic command names to save you typing -- but it is still a command line interface, and that means you interact with the operating system by typing in commands. File names can be long, directory hierarchies can be deep, and this can mean you have to type a lot to specify the file you want or change to current working directory. Not only that, but you have to remember the names of files and directories you type in. The TAB key gives you a way to enter with commands with less typing and less memorization.

Go back to the Shortcuts directory:

$ cd
$ cd Shortcuts

Now enter the following:

$ hostname > file1
$ cal > file2
$ whoami > different-file
$ date > other-file
$ cal > folio5

Now type the following, without hitting the Return key:

$ cat oth <Tab>

What happened? Linux completed the name "other-file" for you! The Tab key is your way of telling Linux to finish the current word you are typing, if possible. Because there is only one file in the directory whose name begins with "oth", when you hit the Tab key Linux is able to complete the name.

Hit Return (if you haven't already) to enter the cat command. Now try

$ cat d <Tab>

As you would expect, Linux completes the name "different-file"

What if you enter

$ cat fi <Tab>

Notice Linux completes as much of the name as possible. You can now enter a "1" or a "2" to finish it off.

But what if you forget what the options are? What if you can't remember if you created "file1" and "file2" or if you created "fileA" and fileB"?

With the comman line showing this:

$ cat file

hit the Tab key twice. Aha! Linux shows you the possible choices for completing the word.

Try

$ cat f <Tab>

The Tab will not add anything -- the command line will still read

$ cat f

Now type the letter o followed by a Tab -- once you add the o there is only one possible completion -- "folio".

Now enter the following:

$ cat directory1/directory2/directory3/directory4/file3

That's kind of a painful to type.

Now type the following without entering Return:

$ ls dir <Tab>

Nice! As you would expect, Linux completes the name of the directory for you. This is because there is only one file in the Shortcuts directory whose name begins with "dir"

Hit Return and Linux will tell you that directory1 contains directory2.

Now type this:

$ ls dir <Tab>

and before you hit return type another d followed by another Tab. Your command line should now look like this:

$ ls directory1/directory2/

If you hit Return, Linux will tell you that directory2 contains directory3.

Now try this:

$ ls dir <Tab>

then type another d followed by <Tab> then another d followed by tab. Don't hit Return yet. Your command line should look like this:

$ ls directory1/directory2/directory3/

Don't hit Return yet. Now type the letter f followed by a Tab. What do you think should happen?

Step 3 -- The Exclamation Point

Hitting the up arrow key is a nice way to recall previously-used commands, but it can get tedious if you are trying to recall a command you entered a while ago -- hitting the same key 30 times is a good way to make yourself feel like an automaton. Fortunately, linux offers a couple of other ways to recall previous commands that can be useful.

Go back to the Shortcuts directory

$ cd ~/Shortcuts

and enter the following:

$ hostname
$ cal
$ date
$ whoami

Now enter this:

$ !c

and hit return.

What happened? Now try

$ !h

and hit return.

The exclamation point ("bang" to Americans, "shriek" to some Englishmen I've worked with) is a way of telling linux you want to recall the last command which matches the text you type after it. So "!c" means recall the last command that starts with the letter c, the "cal" command in this case. You can enter more than one character after the exclamation point in order to distinguish between commands. For example if you enter

$ cd ~/Shortcuts
$ cat file1
$ cal
$ !c

the last command will redo the "cal" command. But if you enter

$ cat file1
$ cal
$ !cat

the last command re-executes the "cat" command.

Step 4 -- Ctrl-r

One problem with using the exclamation point to recall a previous command is that you can feel blind -- you don't get any confirmation about exactly which command you are recalling until it has executed. Sometimes you just aren't sure what you need to type after the exclamation point to get the command you want.

Typing Ctrl-r (that's holding down the Ctrl key and typing a lower case r) is another way to repeat previous commands without having to type the whole command, and it's much more flexible than the bang. The "r" is for "reverse search" and what happens is this. After you type Ctrl-r, start typing the beginning of a previously entered command -- linux will search, in reverse order, for commands that match what you type. To see it in action, type in the following commands (but don't hit <Enter> after the last one):

$ cd ~/Shortcuts
$ cat file1
$ cat folio5
$ cal
$ Ctrl-r cat

You should see the following on your command line:

(reverse-i-search)`cat': cat folio5

Try playing with this now. Type in " fi" (that's a space, an "f" and an "i") -- did the command shown at the prompt change? Now hit backspace four times.

Now enter a right or left arrow key and you will find yourself editing the matching command. This is one you have to play around with a bit before you understand exactly what it is doing. So go ahead and play with it.

Step 5 -- history

Now type

$ history

and hit return.

Cool, huh? You get to see all the commands you have entered (probably a maximum of 1000.) You can also do something like

$ history | grep cal

to get all the commands with the word "cal" in them. You can use the mouse to cut and paste a previous command, or you can recall it by number with the exclamation point:

$ !874

re-executes the command number 874 in your history.

For more information about what you can do to recall previous commands, check out http://www.thegeekstuff.com/2011/08/bash-history-expansion/

Step 6 -- Ctrl-t

I am just including this because to me it is a fun piece of linux trivia. I don't find it particularly useful. Type

$ cat file1

and hit <Return>. Now hit the up arrow key to recall this command and hist the left arrow key twice so the cursor is on the "e" of "file1". Now hit Ctrl-t (again, hold down the control key and type a lower case t.) What just happened? Try hitting Ctrl-t a couple more times. That's right -- it transposes two characters in the command line -- the one the cursor is on and the one to its left. Also, it moves the cursor to the right. Frankly, it takes me more time to think about what is going to happen if I type Ctrl-t than it takes me to delete some characters and retype them in the correct order. But somewhere out there is a linux black belt who gets extra productivity out of this shortcut.

Step 7 -- The alias command

Another nice feature of linux is the alias command. If there is a command you enter a lot you can define a short name for it. For example, we have been typing "cat folio5" a lot in this tutorial. You must be getting sick of typing "cat folio5". So enter the following:

$ alias cf5='cat folio5'

Now type

$ cf5

and hit return. Nice -- you now have a personal shortcut for "cat folio5". I use this for the ssh commands:

$ alias gogl='ssh -Y jeisenl@pitzer.osc.edu'

I put this in the .bash_aliases file on my laptop so that it is always available to me.

Classroom Project Resource Guide

This document includes information on utilizing OSC resources in your classroom effectively.


Request a Classroom Project

Classroom projects will not be billed under the Ohio academic fee structure; all fees will be covered by billing credits from OSC.

Please submit a new project request for a classroom project. You will request a $500 budget; OSC will provide a $500 billing credit. If additional budgets and credits are needed, contact us at OSC Help. We require a class syllabus; this will be uploaded on the last screen before you submit. 

Access

We suggest that students consider connecting to our OnDemand portal to access the HPC resources. All production supercomputing resources can be accessed via that website, without having to worry about client configuration. We have a guide for new students to help them figure out the basics of using OSC; included are basics on getting connected, HPC system structure, file transfers, and batch systems.

Resources

We currently have two production clusters, Pitzer, and Owens, with Nvidia GPUs available. All systems have "debug" queues, during typical business hours, that allow small jobs of less than 1 hour to start much quicker than they might otherwise.

If you need to reserve access to particular resources, please contact OSC Help, preferably with at least two weeks lead time, so that we can put in the required reservations to ensure resources are available during lab or class times.

Software

We have a list of supported software, including sample batch scripts, in our documentation. If you have specific needs that we can help with, let OSC Help know.

If you are using Rstudio, please see this webpage

If you are using Jupyter, please see the page Using Jupyter for Classroom.

Account Maintenance

Our classroom information guide will instruct you on how to get students added to your project using our client portal. For more information, see the documentation. You must also add your username as an authorized user

Homework Submissions

We can provide you with project space to have the students submit assignments through our systems. Please ask about this service and see our how-to. We typically grant 1-5 TB for classroom projects.

Support

Help can be found by contacting OSC Help weekdays, 9AM to 5PM (1-800-686-6472 or 614-292-1800).
Fill out a request online. 

We update our web pages to show relevant events at the center (including training) and system notices on our main page (osc.edu). We also provide important information in the “message of the day” (visible when you log in). Furthermore, you can receive notices by following @HPCNotices on Twitter. 

Helpful Links

FAQ: http://www.osc.edu/supercomputing/faq

Main Supercomputing pages: http://www.osc.edu/supercomputing/

 

 

 

Supercomputer: 

Using Jupyter for Classroom

OSC provides an isolated and custom Jupyter environment for each classroom project that requires Jupyter Notebook or JupyterLab. The interface can be accessed at class.osc.edu. To set up this interface, please apply for a classroom project that is unique for the course. More details on the classroom project can be found in our classroom project guide. The custom Jupyter environment for the course will be tied to this project ID. Please inform us if you have additional requirements for storage. Once we get the information, we will provide you a course ID (which is commonly the course ID provided by PI + school code, e.g. PHYS280_WIT) and add your course to the interface.

Before testing the interface, please follow the instructions below to create a Jupyter environment for the course and send us with the full path of the environment. Once we update the interface, you can login to class.osc.edu and start testing the course. After login, you will see several Apps listed. Pick Jupyter App and that will take you to the Jupyter Job submission page. Please pick your course from the drop-down menu under the Class materials and the number of hours and cores needed.

Set up a Jupyter environment for class

A Jupyter environment for class is created using Python 3 venv module, and it is shared with students. The environment created is an isolated Python environment, meaning that system Python packages and user's local packages become inaccessible when you activate this environment. It is easy for PI to manage packages dedicated for class. 

PI can create a Jupyter environment in either own home or project space.

Using project space (HIGHLY recommended)

If project space is used, e.g. /fs/ess/projectID/courseID/jupyter where you want to install Jupyter environment, no permission change is required because students would be added into the project.

Using home space 
If home space is used, PI must make the home directory accessed by students.

For example, a Jupyter environment is created at $HOME/courseID/jupyter. and your classroom project ID is PAS1234. Add execution permission to $HOME and the sub-directories,

mkdir -p $HOME/courseID/jupyter
nfs4_setfacl -a A:g:PAS1234@osc.edu:X $HOME
nfs4_setfacl -a A:g:PAS1234@osc.edu:X $HOME/courseID

Create a virtual environment

Login to Owens or Pitzer at OSC (see Getting Connected). In a terminal, assuming that the course ID is  PHYSIC6820 and project ID is PAS1234, run the following commands to create a virtual environment in project space

~support/classes/tools/create_class_venv --prompt "PHYSIC6820" /fs/ess/PAS1234/PHYSIC6820/jupyter 

Initiate the environment

After creating the virtual environment, activate the environment:

source /fs/ess/PAS1234/PHYSIC6820/jupyter/bin/activate # use activate.csh if you are in a C shell

Then install Jupyter Notebook and python packges for class, e.g. numpy:

(PHYSIC6820)$ pip install jupyterlab jupyter-console qtconsole ipywidgets
(PHYSIC6820)$ pip install numpy
(PHYSIC6820)$ deactivate

The command deactivate is not required if you continue testing and using packages installed in the Jupyter environment. 

Once PI complete setting up a Jupyter environment, please update us with the full path of the virtual environment in order to complete the interface for classroom.  Once the information is recevied, we will update the interfrace, and you can login to class.osc.edu to test your Jupyter classroom.

Install more packages

PI can install more package anytime in a terminal 

source /fs/ess/PAS1234/PHYSIC6820/jupyter/bin/activate # use activate.csh if you use C shell
(PHYSIC6820)$ pip install pkg1 pkg2 ...
(PHYSIC6820)$ deactivate

or in a Jupyter notebookScreen Shot 2020-08-20 at 10.09.48 AM.png

Enable PDF exporter

To export a notebook as PDF file, it needs complete Tex environment and Pandoc binary (not pandoc python package). The former is taken care of by our system module now. For the latter PI needs to install pandoc in a class Jupyter environment:

%%bash
cd $TMPDIR
wget https://github.com/jgm/pandoc/releases/download/2.10.1/pandoc-2.10.1-linux-amd64.tar.gz
tar xf pandoc-2.10.1-linux-amd64.tar.gz -C $VIRTUAL_ENV --strip=1

Enable local package access (optional)

By default this Jupyter environment is an isolated Python environment. Anyone launches python from this environment can only access packages installed inside unless PYTHONPATH is used. PI can change it by setting include-system-site-packages = true in /PATH/TO/JUPYTER/VENV/pyvenv.cfg. This will allows students to access packages in home directory ~/.local/lib/pythonx.x/site-packages ,and install packages via pip install –user.

Workspace

When a class session starts, we create a classroom workspace under PI's and students' home space: $HOME/osc_classes/courseID, and launch Jupyter at the workspace. The root /  will appear in the landing page (Files) but everything can be found in $HOME/osc_classes/courseID on OSC clusters.

Shared Access

Share class material

PI can share class material from home or project space with students. For example, you have a directory called modules in project space /fs/ess/projectID/courseID/modules  (source) that contains the class material. When a student launch a Jupyter session, the diretory will be copied to the student's worksapce $HOME/osc_classes/courseID (destinatation). The student will see the directory modules at the landing page:

Screen Shot 2020-08-17 at 11.02.26 AM.png

PI can add files to the material source directory. New files will be copied to the destination everytime when a new Jupyter session starts. 

Once you decide the location for the material, please update us with the full path of the material directory.

Access student workspace

PI and TAs can access a student's workspace with limited permissions. First, PI sends us a request with the information including PI's and TAs' OSC accounts. After a student launches a class session, you can access known files and directories in the student's workspace. For example, you cannot explore the student's workspace

ls /users/PZS1234/student1/osc_classes/courseID
ls: cannot open directory /users/PZS1234/student1/osc_classes/courseID: Permission denied

but you can access a known file or directory in the workspace

ls /users/PZS1234/student1/osc_classes/courseID/homework
     

    Using nbgrader for Classroom

    Using ngbrader in Jupyter

    Install nbgrader

    You can install nbgrader in a notebook:

    1. Launch a Juypter session from class.osc.edu
    2. Open a new notebook 
    3. To Install nbgrader, run:
    !pip install nbgrader
    !jupyter nbextension install --sys-prefix --py nbgrader --overwrite 
    !jupyter nbextension enable --sys-prefix --py nbgrader 
    !jupyter serverextension enable --sys-prefix --py nbgrader
    

    To check the installed extensions, run

    !jupyter nbextension list
    

    There are six enabled extensionsScreen Shot 2020-08-21 at 11.31.36 PM.png

    Configure nbgrader

    In order to upload and collect assignments, nbgrader requires a exchange directory with write permissions for everyone. For example, to create a directory in project space, run:

    %%bash
    mkdir -p /fs/ess/projectID/courseID/exchange
    chmod a+wx /fs/ess/projectID/courseID/exchange

    Then get your cousre ID for configuratin. In a notebook, run:

    %%bash
    echo $OSC_CLASS_ID
    

    Finally create the nbgrader configuration at the root of the workspace. In a notebook, run 

    %%file nbgrader_config.py
    c = get_config()
    c.CourseDirectory.course_id = "courseID"     # it must be the value of $OSC_CLASS_ID
    c.Exchange.root = "/fs/ess/projectID/courseID/exchange"
    c.Exchange.timezone = 'EST'
    

    Once the file is created, you can launch a new Jupyter session then start creating assignments. For using nbgrader, please refer the nbgrader documents.

    Access assignments

    To let students access the assignments, students need to have the following configuration file in the root of their workspace:

    %%file nbgrader_config.py
    c = get_config()
    c.Exchange.root = "/fs/ess/projectID/courseID/exchange"
    
       

      HOWTO

      Our HOWTO collection contains short tutorials that help you step through some of the common (but potentially confusing) tasks users may need to accomplish, that do not quite rise to the level of requiring more structured training materials. Items here may explain a procedure to follow, or present a "best practices" formula that we think may be helpful.

      Service: 

      HOW TO: Look at requested time accuracy using XDMoD

      The XDMoD tool at xdmod.osc.edu can be used to get an overview of how accurate the requested time of jobs are with the elapsed time of jobs.

      One way of specifying a time request is:

      #SBATCH --time=xx:xx:xx

      The elapsed time is how long the job ran for before completing. This can be obtained using the  sacct command.

      $ sacct -u <username> --format=jobid,account,elapsed

      It is important to understand that the requested time is used when scheduling a submitted job. If a job requests a time that is much more than the expected elapsed time, then it may take longer to start because the resources need to be allocated for the time that the job requests even if the job only uses a small portion of that requested time.

      This allows one to view the requested time accuracy for an individual job, but XDMoD can be used to do this for jobs submitted in over a time range.

      First, login to xdmod.osc.edu, see this page for more instructions.

      https://www.osc.edu/supercomputing/knowledge-base/xdmod_tool

      Then, navigate to the Metric Explorer tab.

      Look for the Metric Catalog on the left side of the page and expand the SUPREMM options. Select Wall Hours: Requested: Per Job and group by None.

      walltime_acc_metric_tab.png

      This will now show the average time requested.

      The actual time data can be added by navigating to Add Data -> SUPREMM -> Wall Hours: Per Job.

      walltime_acc_add_data.png

      walltime_acc_select_walltime.png

      This will open a new window titled Data Series Definition, to change some parameters before showing the new data. In order to easily distinguish between elapsed and requested time, change the Display Type to Bar, then click add to view the new data.

      walltime_add_data_settings.png

      Now there is a line which shows the average requested time of jobs, and bars which depict the average elapsed time of jobs. Essentialy, the closer the bar is to the line, without intersecting the line, the more accurate the time predicition. If the bar intersects the line, then it may indicate the there was not enough time requested for a job to complete, but remember that these values are averages.

      walltime_acc_final_zoom.png

      One can also view more detailed information about these jobs by clicking a data point and using the Show raw data option.

      wall_acc_select_datapoint.png

      In order to have the Show raw data option, one may need to use the Drilldown option first to sort the jobs in that list by use or another metric.

      wall_acc_show_raw_data.png

      Supercomputer: 

      HOWTO: Add python packages using the conda package manager

      While our Python installations come with many popular packages installed, you may come upon a case where you need an addiditonal package that is not installed.  If the specific package you are looking for is available from anaconda.org (formerlly binstar.org) you can easily install it and required dependencies by using the Conda package manager.

      To be able to install a package using the Conda package manager:

      • You must use an Anaconda distribution of Python:
        • Load an appropriate python module, for example module load python/3.6-conda5.2 on Pitzer.
        • Run conda --version to check that a version of Anaconda was successfully loaded.
      • Package should be available through anaconda.org and/or using pip (see next section).

      If you would like to freeze a distribution so that you can control how often you update Anaconda, please send us a help request at oschelp@osc.edu.

      Procedure

      The following steps are an example of how to set up a python environment and install packages to a local directory using Conda. We use the name  local for the environment, but you may use any other name.

      Load proper python module

      module load python/3.6-conda5.2
      

      Create python installation to local directory

      Three alternative create commands are listed.  These cover the most common cases.

      Clone base environment:

      If you want to clone the full base python environment, you may use the following create command:

      conda create -n local --clone base
      
      Create New Environment:

      The following will create a minimal python installation without any extraneous packages:

      conda create -n local
      
      Create New Environment with specific packages:

      You can augment the command above by listing specific packages you would like installed into the environment. For example, the following will create a minimal python installation with only the specified packages (in this case, numpy and babel):

      conda create -n local numpy babel
      

      By default, conda will install the newest versions of the packages it can find. Specific versions can be specified by adding =<version> after the package name. For example, the following will create a python installation with Python version 2.7 and NumPy version 1.16:

      conda create -n local python=2.7 numpy=1.16
      

       

      To verify that a clone has been created, use the command

      conda info -e

      For additional conda command documentation see https://docs.conda.io/projects/conda/en/latest/commands.html#conda-general-commands

      Activate environment

      Before the created environment can be used, it must be activated.

      For the bash shell:

      source activate local
      

      On newer versions of Anaconda on the Owens cluster you may also need to perform the removal of the following packages before trying to install your specific packages:

      conda remove conda-build
      conda remove conda-env
      

      Install packages

      To install additional packages, use the conda install command. For example, to install the yt package:

      conda install yt
      

      By default, conda will install the newest version if the package that it can find. Specific versions can be specified by adding =<version> after the package name. For example, to install version 1.16 of the NumPy package:

      conda install numpy=1.16
      
      If there are errors on this step you will need to resolve them before continuing.

      Test python package

      Now we will test our installed python package by loading it in python and checking its location to ensure we are using the correct version. For example, to test that NumPy is installed correctly, run

      python -c "from __future__ import print_function; import numpy; print(numpy.__file__)"
      

      and verify that the output generally matches

      $HOME/.conda/envs/local/lib/python3.6/site-packages/numpy/__init__.py
      

      To test installations of other packages, replace all instances of numpy with the name of the package you installed.

      Remember, you will need to load the proper version of python before you go to use your newlly installed package.  Packages are only installed to one version of python.

      Install your own python modules

      If the method using conda above is not working or if you prefer, you can consider installing python modules from the source. Please read HOWTO: install your own python modules.

      But I use virtualenv and/or pip!

      See the comparison to these package management tools here:

      https://docs.conda.io/projects/conda/en/latest/commands.html#conda-vs-pip-vs-virtualenv-commands

      Pip installations are supported:

      module load python
      module list # check which python you just loaded
      pip install --upgrade --user PACKAGE # where PACKAGE is a valid package name
      

      If binaries exist in a pip installation they are usually installed in:

      $HOME/.local/bin
      

      The one issue with this approach is portability with multiple Python modules.  If you plan to stick with a single Python module, then this should not be an issue.  However, if you commonly switch between Python 2 and Python 3, then be aware of the potential trouble in using the same installation location for all Python versions.

      Virtualenv may also work, but is not under active testing at OSC.

      Further Reading:

      Conda Test Drive: https://conda.io/docs/test-drive.html 

      Supercomputer: 

      HOWTO: Install Tensorflow locally

      This documentation describes how to install tensorflow package locally in your $HOME space.

      Load python module

      module load python/3.6-conda5.2
      

      Clone python installation to local directory

      Three alternative create commands are listed.  These cover the most common cases:

      conda create -n local --clone="$PYTHON_HOME"
      

      This will clone the entire python installation to ~/envs/local directory. The process will take several minutes.

      conda create -n local
      

      This will create a local python installation without any packages. If you need a small number of packages, you may choose this option.

      conda create -n local python={version} anaconda
      

      If you like to install a specific version of python, you can specify it with "python" option. For example, you can use "python=3.6" for version 3.6.

      To verify that a clone has been created, use the command

      conda info -e

      For additional conda command documentation see https://conda.io/docs/commands.html

      Activate clone environment

      For the bash shell:

      source activate local
      

      On newer versions of Anaconda on the Owens cluster you may also need to perform the removal of the following packages before trying to install your specific packages:

      conda remove conda-build
      conda remove conda-env
      
      

      Install package

      Install the latest version of tensorflow that is gpu compatible.

      pip install tensorflow-gpu
      
      If there are errors on this step you will need to resolve them before continuing.

      Test python package

      Now we will test tensorflow package by loading it in python and checking its location to ensure we are using the correct version.

      python -c "import tensorflow;print (tensorflow.__file__)"
      

      Output:

      $HOME/.conda/envs/local/lib/python2.7/site-packages/tensorflow/__init__.py
      
      Remember, you will need to load the proper version of python before you go to use your newly installed package.  Packages are only installed to one version of python.

      Install your own python modules

      If the method using conda above is not working or if you prefer, you can consider installing python modules from the source. Please read HOWTO: install your own python modules.

       

      Supercomputer: 

      HOWTO: Install your own python modules

      While we provide a number of Python modules, you may need a module we do not provide. If it is a commonly used module, or one that is particularly difficult to compile, you can contact OSC Help for assistance, but we have provided an example below showing how to build and install your own Python modules, and make them available inside of Python. Note, these instructions use "bash" shell syntax; this is our default shell, but if you are using something else (csh, tcsh, etc), some of the syntax may be different.

      Please consider using conda python package manager before you try to build python using the method explained here. We have an instruction on conda here.

      Gather your materials

      First, you need to collect up what you need in order to do the installation. To keep things tidy, we will do all of our work in $HOME/local/src . You should make this directory now.

      mkdir -p $HOME/local/src
      

      Now, we will need to download the source code for the module we want to install. In our example, we will use "NumExpr", a module we already provide in the system version of Python. You can either download the file to your desktop, and then upload it to OSC, or directly download it using the wget utility (if you know the URL for the file).

      cd ~/local/src
      wget http://numexpr.googlecode.com/files/numexpr-2.0.1.tar.gz
      

      Now, extract the downloaded file. In this case, since it's a "tar.gz" format, we can use tar to decompress and extract the contents.

      tar xvfz numexpr-2.0.1.tar.gz
      

      You can delete the downloaded archive now, if you wish, or leave it around should you want to start the installation from scratch.

      Build it!

      Environment

      To build the module, we will want to first create a temporary environment variable to aid in installation. We'll call it "INSTALL_DIR".

      export INSTALL_DIR=${HOME}/local/numexpr/2.0.1
      

      I am following, roughly, the convention we use at the system level. This allows us to easily install new versions of software without risking breaking anything that uses older versions. We have specified a folder for the program (numexpr), and for the version (2.0.1). Now, to be consistent with python installations, we're going to create a second temporary environment variable, which will contain the actual installation location.

      export TREE=${INSTALL_DIR}/lib/python2.7/site-packages
      

      Now, make the directory tree.

      mkdir -p $TREE
      

      Compile

      To compile the module, we should switch to the GNU compilers. The system installation of Python was compiled with the GNU compilers, and this will help avoid any unnecessary complications. We will also load the Python module, if it hasn't already been loaded.

      module swap intel gnu
      module load python
      

      Now, build it. This step may vary a bit, depending on the module you are compiling. You can execute python setup.py --help to see what options are available. Since we are overriding the install path to one that we can write to, and that fits our management plan, we need to use the --prefix option.

      python setup.py install --prefix=$INSTALL_DIR
      

      Make it usable

      At this point, the module is compiled and installed in ~/local/numexpr/2.0.1/lib/python2.7/site-packages . Occasionally, some files will be installed in ~/local/numexpr/2.0.1/bin as well. To ensure Python can locate these files, we need to modify our environment.

      Manual

      The most immediate way - but the one that must be repeated every time you wish to use the module - is to manually modify your environment. If files are installed in the "bin" directory, you'll need to add it to your path. As before, these examples are for bash, and may have to be modified for other shells. Also, you will have to modify the directories to match your install location.

      export PATH=$PATH:~/local/numexpr/2.0.1/bin
      

      And, for the python libraries:

      export PYTHONPATH=$PYTHONPATH:~/local/numexpr/2.0.1/lib/python2.7/site-packages
      

      Hardcode it

      We don't really recommend this option, as it is less flexible, and can cause conflicts with system software. But, if you want, you can modify your .bashrc (or similar file, depending on your shell) to set these environment variables automatically. Be extra careful; making a mistake in .bashrc (or similar) can destroy your login environment in a way that will require a system administrator to fix. To do this, you can copy the lines above modifying $PATH and $PYTHONPATH into .bashrc. Remember - test them interactively first! If you destroy your shell interactively, the fix is as simple as logging out and then logging back in. If you break your login environment, you'll have to get our help to fix it.

      Make a module (recommended!)

      This is the most complicated option, but it is also the most flexible, as you can have multiple versions of this particular software installed, and specify at run-time which one to use. This is incredibly useful if a major feature changes that would break old code, for example. You can see our tutorial on writing modules here, but the important variables to modify are, again, $PATH and $PYTHONPATH . You should specify the complete path to your home directory here, and not rely on any shortcuts like ~ or $HOME .  Below is a modulefile written in Lua:

      If you are following the tutorial on writing modules, you will want to place this file in $HOME/local/share/modulefiles/numexpr/2.0.1.lua :

      -- This is a Lua modulefile, this file 2.0.1.lua can be located anywhere
      -- But if you are following a local modulefile location convention, we place them in
      -- $HOME/local/share/modulefiles/
      -- For numexpr we place it in $HOME/local/share/modulefiles/numexpr/2.0.1.lua
      -- This finds your home directory
      local homedir = os.getenv("HOME")
      prepend_path("PYTHONPATH", 
      pathJoin(homedir, "/local/numexpr/2.0.1/lib/python2.7/site-packages"))
      prepend_path(homedir, "local/numexpr/2.0.1/bin"))
      

       

      Once your module is created (again, see the guide), you can use your python module simply by loaded the software module you created.

      module use $HOME/local/share/modulefiles/
      module load numexpr/2.0.1
      
      Supercomputer: 
      Service: 

      HOWTO: Collect performance data for your program

      This page outlines ways to generate and view performance data for your program using tools available at OSC.

      Intel Tools

      This section describes how to use performance tools from Intel. Make sure that you have an Intel module loaded to use these tools.

      Intel VTune

      Intel VTune is a tool to generate profile data for your application. Generating profile data with Intel VTune typically involves three steps:

      1. Prepare the executable for profiling.

      You need executables with debugging information to view source code line detail: re-compile your code with a -g  option added among the other appropriate compiler options. For example:

      mpicc wave.c -o wave -g -O3
      

      2. Run your code to produce the profile data.

      Profiles are normally generated in a batch job. To generate a VTune profile for an MPI program:

      mpiexec <mpi args> amplxe-cl <vtune args> <program> <program args>

      where <mpi args> represents arguments to be passed to mpiexec, <program> is the executable to be run, <vtune args> represents arguments to be passed to the VTune executable amplxe-cl, and <program args> represents arguments passed to your program.

      For example, if you normally run your program with mpiexec -n 12 wave_c, you would use

      mpiexec -n 12 amplxe-cl -collect hotspots -result-dir r001hs wave_c

      To profile a non-MPI program:

      amplxe-cl <vtune args> <program> <program args>

      The profile data is saved in a .map file in your current directory.

      As a result of this step, a subdirectory that contains the profile data files is created in your current directory. The  subdirectory name is based on the -result-dir argument and the node id, for example, r001hs.o0674.ten.osc.edu.

      3. Analyze your profile data.

      You can open the profile data using the VTune GUI in interactive mode. For example:

      amplxe-gui r001hs.o0674.ten.osc.edu

      One should use an OnDemand VDI (Virtual Desktop Interface) or have X11 forwarding enabled (see Setting up X Windows). Note that X11 forwarding can be distractingly slow for interactive applications.

      Intel ITAC

      Intel Trace Analyzer and Collector (ITAC) is a tool to generate trace data for your application. Generating trace data with Intel ITAC typically involves three steps:

      1. Prepare the executable for tracing.

      You need to compile your executbale with -tcollect  option added among the other appropriate compiler options to insert instrumentation probes calling the ITAC API. For example:

      mpicc wave.c -o wave -tcollect -O3
      

      2. Run your code to produce the trace data.

      mpiexec -trace <mpi args> <program> <program args>

      For example, if you normally run your program with mpiexec -n 12 wave_c, you would use

      mpiexec -trace -n 12 wave_c

      As a result of this step, .anc, .f, .msg, .dcl, .stf, and .proc files will be generated in your current directory.

      3. Analyze the trace data files using Trace Analyzer

      You will need to use traceanalyzer to view the trace data. To open Trace Analyzer:

      traceanalyzer /path/to/<stf file>

      where the base name of the .stf file will be the name of your executable.

      One should use an OnDemand VDI (Virtual Desktop Interface) or have X11 forwarding enabled (see Setting up X Windows) to view the trace data. Note that X11 forwarding can be distractingly slow for interactive applications.

      Intel APS

      Intel's Application Performance Snapshot (APS) is a tool that provides a summary of your application's performance . Profiling HPC software with Intel APS typically involves four steps:

      1. Prepare the executable for profiling.

      Regular executables can be profiled with Intel APS. but source code line detail will not be available. You need executables with debugging information to view source code line detail: re-compile your code with a -g  option added among the other approriate compiler options. For example:

      mpicc wave.c -o wave -tcollect -O3
      

      2. Run your code to produce the profile data directory.

      Profiles are normally generated in a batch job. To generate profile data for an MPI program:

      mpiexec -trace <mpi args> <program> <program args>

      where <mpi args> represents arguments to be passed to mpiexec, <program> is the executable to be run and <program args> represents arguments passed to your program.

      For example, if you normally run your program with mpiexec -n 12 wave_c, you would use

      mpiexec -n 12 wave_c

      To profile a non-MPI program:

      aps <program> <program args>

      The profile data is saved in a subdirectory in your current directory. The directory name is based on the date and time, for example, aps_result_YYYYMMDD/

      3. Generate the profile file from the directory.

      To generate the html profile file from the result subdirectory:

      aps --report=./aps_result_YYYYMMDD
      

      to create the file aps_report_YYYYMMDD_HHMMSS.html.

      4. Analyze the profile data file.

      You can open the profile data file using a web browswer on your local desktop computer. This option typically offers the best performance.

      ARM Tools

      This section describes how to use performance tools from ARM.

      ARM MAP

      Instructions for how to use MAP is available here.

      ARM DDT

      Instructions for how to use DDT is available here.

      ARM Performance Reports

      Instructions for how to use Performance Reports is available here.

      Other Tools

      This section describes how to use other performance tools.

      HPC Toolkit

      Rice University's HPC Toolkit is a collection of performance tools. Instructions for how to use it at OSC is available here.

      TAU Commander

      TAU Commander is a user interface for University of Oregon's TAU Performance System. Instructions for how to use it at OSC is available here.

      Supercomputer: 
      Service: 

      HOWTO: Connect to OSC services using OSC Connect

       


      NOTE:  OSC Ondemand now provides in-browser connectivity via VNC.  OSC Connect is no longer required nor recommended.  We recommend contacting OSC Help if you need assistance using the in-browser capabilities or if there is other functionality that is forcing you to still use OSC Connect.  This page is still available primarily for reference.


      To connect to OSC services, a secure tunnel to a session is required. This can be done relatively simply in OSX and Linux by using the SSH functionality built into the system, but Windows users have had to configure and use third party applications like PuTTY or Java to access secure resources at OSC. OSC Connect is a native windows application written in C# and compiled for .NET 2.0, providing preconfigured management of secure tunnel connections for Windows users, as well as providing a launcher for secure file transfer, VNC, terminal, and web based services. 

      OSC Connect is supported on Windows versions from Windows XP through Windows 10.

      We've created a brief video:

       

       

      Getting Started

      1. Download the latest release of OSC Connect from here: Latest Build
        • Click OSCConnect.exe  and download. Use "Save link as" to download this file to a folder of your choice. 
        • See here for all the previous releases. 
      2. Double-Click the OSCConnect.exe icon to run the application. In the current state, OSC Connect is entirely deployed by a single executable file; no further installation is required.   ​
      3. When you run OSC Connect, a temporary folder named "ConnectFiles" with four additional files will be created. These are required for proper functionality of the application. Please ensure that these files are permitted by your IT administrator:
        • plink.exe is the command-line version of PuTTY used by the application to create the secure connection to OSC resources.
        • putty.exe  is the GUI application of PuTTY used to provide terminal emulation remote console connections to OSC resources.
        • vncviewer.exe is the VNC viewer client used to view a remote desktop session.
        • WinSCP.exe  is the SFTP client used for file transfer.
      4. Once your connections to OSC services as well as the OSC Connect app is closed, the temporary folder named "ConnectFiles" will be removed automatically. 

      OSC Connect Graphical User Interface

      After you double-click the  OSCConnect.exe  icon oscconnect_icon , the application graphical user interface is shown as below:

       interface

      • Network Status: it indicates which OSC cluster you will be connected to. The option can be changed in "Settings". 

      • Settings: it provides several configuration options to modify the behavior of the application. 

        • Connection Settings: use this dropdown to select the default host/cluster. Selecting a server here will change the endpoint for tunneling, sftp connections, console connections, and connectivity checking.

        • System Settings

          • Detect Clipboard Activity: when this option is enabled, the application will detect valid data on the Windows clipboard and populate the application. ( Default: Off )

          • Check for New Versions: when this option is enabled, the application will check for version updates. (Default: on)

        • Automation Settings:

          • Save User Credentials: when this option is enabled, it allows the application to remember the user when the application is reopened. This saves the user credentials to the user settings using DPAPI Encryption. Passwords are decrypted only by current the Windows user account. ( Default: Off )

          • Launch Tunnel On Import: when this option is enabled, the tunnel will automatically connect when the application detects a valid clipboard string and the user credentials have been entered. ( Default: On )

        • VNC Settings

      • ​About: it provides additional information about OSC Connect.

      After you provide your OSC Credentials, i.e. your OSC HPC username and password, more functionalities are available as shown below:

      gui2

      • SFTP: it uses WinSCP as the SFTP client to securely connect to the OSC file system over SFTP.
      • Console: it uses PuTTY to provides you shell access to OSC cluster.

      In addition, Session Type is provided such that advanced users can connect to a running session mannually.

      For most users, you can ingore Session Type and are able to connect to a running session through OSC OnDemand using the configuration generated after your session starts. See "Connecting to a Session" for more details. 

      Connecting to a Session

      The OSC Connect application can be used to connect to a running session launched through OSC OnDemand.

      OSC Connect does not have the ability to launch a new session
      1. Navigate to OSC Demand to create a new session under either "Desktops" or "Desktop Apps". 
        • More information on how to use OSC OnDemand can be found here
      2. Once your session is running, you have two options to connect to the session: "noVNC Instructions" and "Native Client Instructions". Click "Native Client Instructions" to use OSC Connect. 
      3. Follow the steps below to connect to the running session:
        • Step 1: Download the latest OSC Connect if you don't already have it. See "Getting Started" for more details. 
        • Step 2: Launch OSC Connect. 
        • Step 3: Click - osc://xxxxx . osc://xxxxx is a custom URI scheme that is registered when you launch the application. Simply click the link to populate the configuration information and connect to your running session. If OSCConnect.exe is not running when you click the URI, the OSC Connect application will be popped up. Enter your OSC HPC username and password and you will be able to connect to the session by clicking the "Connect" button.
        • For advanced users, you can skip step 3 and connect to a running session mannually. Depending on the type of application, you can choose either "iHPC VNC Desktop" to create a VNC tunnel or "Web Browser Session" to create a browser tunnel. 
      Be sure to launch OSCConnect.exe at least once before you use it to connect to a running session. The initial launch will add a key to your user registry that initializes the URI scheme.  

      FAQ

      I've clicked the osc:// link and nothing happened.

      Be sure to run OSCConnect.exe at least once. The initial launch will add a key to your user registry that initializes the URI scheme. If you move or rename the OSCConnect.exe file, you will need to run the application again manually to update the path in the handler.

      I've received the error "Unable to open helper application. The protocol specified in this address is not valid."

      This issue appears in some earlier versions of Internet Explorer when attempting to launch the application from a Temporary location. Download and run the OSCConnect.exe application, being sure to save the file to a non-temporary location.

       

       

      Supercomputer: 
      Service: 

      HOWTO: Debugging Tips

      This article focuses on debugging strategies for C/C++ codes, but many are applicable to other languages as well.

      Rubber Duck Debugging

      This approach is a great starting point. Say you have written some code, and it does not do what you expect it to do. You have stared at it for a few minutes, but you cannot seem to spot the problem.

      Try explaining what the problem is to a rubber duck. Then, walk the rubber duck through your code, line by line, telling it what it does. Don’t have a rubber duck? Any inanimate object will do (or even an animate one if you can grab a friend).

      It sounds silly, but rubber duck debugging helps you to get out of your head, and hopefully look at your code from a new perspective. Saying what your code does (or is supposed to do) out loud has a good chance of revealing where your understanding might not be as good as you think it is.

      Printf() Debugging

      You’ve written a whole bunch of new code. It takes some inputs, chugs along for a while, and then creates some outputs. Somewhere along this process, something goes wrong. You know this because the output is not at all what you expected. Unfortunately, you have no idea where things are going wrong in the code.

      This might be a good time to try out printf() debugging. It’s as simple as its name implies: simply add (more) printf() statements to your code. You’ve likely seen this being used. It’s the name given to the infamous ‘printf(“here”);’ calls used to verify that a particular codepath is indeed taken.

      Consider printing out arguments and return values to key functions. Or, the results or summary statistics from large calculations. These values can be used as “sanity checks” to ensure that up until that point in the code, everything is going as expected.

      Assertion calls, such as "assert(...)", can also be used for a similar purpose. However, often the positive feedback you get from print statements is helpful in when you’re debugging. Seeing a valid result printed in standard out or a log file tells you positively that at least something is working correctly.

      Debuggers

      Debuggers are tools that can be used to interactively (or with scripts) debug your code. A fairly common debugger for C and C++ codes is gdb. Many guides exist online for using gdb with your code.

      OSC systems also provide the ARM DDT debugger. This debugger is designed for use with HPC codes and is arguably easier to use than gdb. It can be used to debug MPI programs as well.

      Debuggers allow you to interact with the program while it is running. You can do things like read and write variable values, or check to see if/when certain functions are called.

      Testing

      Okay, this one isn’t exactly a debugging strategy. It’s a method to catch bugs early, and even prevent the addition of bugs. Writing a test suite for your code that’s easy to run (and ideally fast) lets you test new changes to ensure they don’t break existing functionality.

      There are lots of different philosophies on testing software. Too many to cover here. Here’s two concepts that are worth looking into: unit testing and system testing.

      The idea behind unit testing is writing tests for small “units” of code. These are often functions or classes. If you know that the small pieces that make up your code work, then you’ll have more confidence in the overall assembled program. There’s an added architecture benefit here too. Writing code that is testable in the first place often results in code that’s broken up into separate logical pieces (google “separation of concerns”). This makes your code more modular and less “spaghetti-like”. Your code will be easier to modify and understand.

      The second concept – system testing – involves writing tests that run your entire program. These often take longer than unit tests, but have the added benefit that they’ll let you know whether or not your entire program still works after introducing a new change.

      When writing tests (both system and unit tests), it’s often helpful to include a couple different inputs. Occasionally a program may work just fine for one input, but fail horribly with another input.

      Minimal, Reproducible Example

      Maybe your code takes a couple hours (or longer…) to run. There’s a bug in it, but every time you try to fix it, you have to wait a few hours to see if the fix worked. This is driving you crazy.

      A possible approach to make your life easier is to try to make a Minimal, Reproducible Example (see this stackoverflow page for information).

      Try to extract just the code that fails, from your program, and also its inputs. Wrap this up into a separate program. This allows you to run just the code that failed, hopefully greatly reducing the time it takes to test out fixes to the problem.

      Once you have this example, can you make it smaller? Maybe take out some code that’s not needed to reproduce the bug, or shrink the input even further? Doing this might help you solve the problem.

      Tools and other resources

      • Compiler warnings – compilers are your friend. Chances are your compiler has a flag that can be used to enable more warnings than are on by default. GNU tools have “-Wall” and “-Wextra”. These can be used to instruct the compiler to tell you about places in the code where bugs may exist.
      • The Practice of Programming by Brian Kernighan and Rob Pike contains a very good chapter on debugging C and C++ programs.
      • Valgrind is a tool that can be used for many types of debugging including looking for memory corruptions and leaks. However, it slows down your code a very sizeable amount. This might not be feasible for HPC codes
      • ASAN (address sanitizer) is another tool that can be used for memory debugging. It is less featureful than Valgrind, but runs much quicker, and so will likely work with your HPC code.

       

      Supercomputer: 
      Service: 

      HOWTO: IME, Cache & Burst Buffer for Scratch File System

      The Owens and Pitzer clusters have access to the DDN Infinite Memory Engine (IME), a fast data tier between the compute nodes and the /fs/scratch file system. IME is a Solid State Disk (SSD) that can act as a cache and burst buffer to improve the performance of the scratch file system.

      While some jobs will benefit from using IME, others will not. You need to understand the workflow of IME well, since sometimes it is not very intuitive. Files must be explicitly imported, synchronized and/or purged.

      When to Use IME

      Benefits of Using IME

      The most obvious reason for using IME is to speed up the I/O in your job. Jobs with heavy I/O may benefit from IME.

      Another reason to use IME is to reduce the I/O load on the scratch file system. A job that writes a large amount of data to /fs/scratch in a short period of time may overload the servers and degrade performance for other users. Even if IME doesn’t improve performance for that job, it may have a system-wide benefit.

      Limitations of IME

      IME should not be used if many small files are being written or if files are opened and closed frequently. Metadata-intensive operations such as these are likely to be slower on IME than on the underlying file system.

      We recommand to test properly before using IME for production. If data is not managed properly (see below) it’s possible to lose data or end up with corrupted files.

      The IME system at OSC is still somewhat new and may not be as stable as the standard file systems.

      IME Concepts

      IME is a user-managed cache available on Owens and Pitzer scratch file systems. Data in /fs/scratch is not automatically loaded into IME. Similarly, data written to IME is not automatically written to /fs/scratch . Here we define some terminology as a means of introducing important concepts.

      resident

      File data is resident in IME if it is present in the IME layer. It may or may not also exist in /fs/scratch . Data is made resident either by explicitly importing it with the IME command-line utility ime-ctl or by directly writing data to the IME.

      clean

      Data resident in IME is clean if it has not been modified since it was last imported or synchronized. Clean data matches the corresponding file data in /fs/scratch unless the file was modified in /fs/scratch . A file typically consists of many fragments, or chunks, some of which may be clean and others dirty.

      dirty

      Data resident in IME is dirty if it has been written or modified and not synchronized. Dirty data are not visible on /fs/scratch and may also not be visible from other IME clients.

      import

      A file in /fs/scratch may be imported into IME using the command-line utility ime-ctl . The file is then resident in IME; the file in /fs/scratch is unchanged. Reading a file through the IME interface without explicitly importing it does not import it, or make it resident in IME, and is typically slower than reading it directly from /fs/scratch .

      synchronize

      Synchronization is the process of writing dirty file data from IME to /fs/scratch . The user must explicitly synchronize an IME-resident file, using the command-line utility ime-ctl , to make it visible on /fs/scratch . There is no automatic synchronization. Exception: If IME gets too full, the system will synchronize and purge older resident data to make space for new data.

      purge

      File data resident in IME remains resident until it is purged. Files should be purged when they are no longer needed in IME, typically at the end of a job. Because IME is small (40TB) compared to the size of /fs/scratch , all users are asked to purge their data from IME promptly unless there is a reason for keeping it resident. Purging a file from IME does not affect the corresponding file in /fs/scratch . Unsynchronized data will be lost if it is purged. The command-line utility ime-ctl is used to purge data. The system may purge resident data if IME gets too full; dirty data is synchronized before purging.

      FUSE

      The FUSE (Filesystem in Userspace) interface is a POSIX mount point at /ime/scratch . The /ime/scratch directory is available only through batch jobs that explicitly request it (see below); it is not mounted on the login nodes. All files in /fs/scratch are visible in /ime/scratch , even when they are not resident in IME.

      Relationship between IME and /fs/scratch

      The interaction between IME and /fs/scratch is complex and sometimes not intuitive. Here are a few points to keep in mind.

      • All files and data on /fs/scratch are visible from IME even if they are not resident in IME.
      • All files in IME are visible from /fs/scratch although data is not available unless it is synchronized. File size may not be correct, showing as 0 for unsynchronized files created and written in IME.
      • File permissions are the same in /ime/scratch and /fs/scratch . If a read-only file is created in IME, it can't be synchronized because the file in scratch can't be written. If a read-only file is imported into IME, it can't be purged from IME.
      • If a file is changed in /fs/scratch while it is resident in IME, results are unpredictable. Data loss or corruption is likely. Purge the file from IME before modifying in /fs/scratch .
      • Although files are not automatically synchronized or purged, the system may take these actions without warning if it becomes necessary to free up space in IME. Dirty data is synchronized before it is purged in this situation.
      • Recall that IME is much smaller than /fs/scratch with capacity of 40TB.
      Warning: Never directly modify a file in /fs/scratch while it is resident in IME.

      How to Access IME

      IME is available at OSC only through batch jobs, not on the login nodes.

      There are three ways to access IME. We consider them in order of increasing programming effort and performance.

      /ime/scratch FUSE filesystem

      The easiest way to access IME is to use the directory /ime/scratch , a POSIX mount point known as the FUSE (Filesystem in Userspace) interface.

      The /ime/scratch directory is mounted on a compute node only when a user job explicitly requests it by adding the :ime specification to a node request:

      #SBATCH --nodes=1 --ntasks=40 --gres=ime
      

      All files in /fs/scratch will appear under /ime/scratch , but that doesn't mean the data is resident in the IME. Accessing files on /fs/scratch through the /ime/scratch directory is slower than accessing them directly.

      See the Use Cases below for examples on how to access IME through the FUSE interface.

      $IMEDIR environment variable

      Every job that requests IME has an environment variable $IMEDIR pointing to the IME counterpart to the $PFSDIR job-temporary scratch directory. That is, if $PFSDIR is /fs/scratch/MYGROUP/myname/jobid , then $IMEDIR is /ime/scratch/MYGROUP/myname/jobid . Since $PFSDIR is deleted at the end of the job, $IMEDIR goes away also.

      MPI-IO API (or libraries built on this API)

      Some of our MPI installations are built using an IME-specific version of ROMIO to provide high performance MPI-IO operations on IME. You must build and run your application using one of the special MPI modules to take advantage of this feature. This is not currently available at OSC.

      IME Command Line Utility

      The ime-ctl command is used to import, synchronize, and purge IME file data as well as check status. It works through the FUSE interface, /ime/scratch . You can include these commands in a batch job with the :ime specification on the nodes= line.

      To manage your IME file data outside of a batch job you should use an interactive batch job:

      salloc -t 1:00:00 --ntasks-per-node=1 --gres=ime srun --pty /bin/bash
      

      Following are some useful ime-ctl options. In all cases the file or directory name may be specified as either an absolute or a relative path and must be located in the /ime/scratch directory.

      Add -R to make an operation recursive: ime-ctl -p -R mydirectory

      help

      ime-ctl -h
      

      Displays help and usage message.

      import

      ime-ctl -i /ime/scratch/filename
      

      Import the entire file, first purging any clean data already resident.

      ime-ctl -i -K /ime/scratch/filename
      

      Import the file data but keep existing clean data during the import operation.

      Note: The import operation by default is nonblocking. Add -b to block on import completion.

      ime-ctl -i -b /ime/scratch/filename
      

      synchronize

      ime-ctl -r /ime/scratch/filename
      

      Synchronize the file data from IME to /fs/scratch , keeping the data in IME.

      Note: The synchronize operation by default is nonblocking. Add -b to block on synchronize completion.

      ime-ctl -r -b /ime/scratch/filename
      

      purge

      ime-ctl -p /ime/scratch/filename
      

      Purge the file data from IME. Unsynchronized data will be lost. The file on /fs/scratch is not affected.

      show fragment status

      ime-ctl -s /ime/scratch/filename
      

      Show fragment status of file data resident in IME. Fragments can be in one of four states: dirty, pending (in the process of synchronizing), clean, deletable (in the process of being purged).

      Use Cases

      This section gives detailed command sequences for some common situations where IME may be used. The examples strongly depend on the software it uses. So, you need to undertand the I/O structure of the software.

      Temporary files

      Temporary files are those that are written and read back in but discarded at the end of a job. In this example we use the FUSE interface with the job-specific $IMEDIR directory. Files are written and read but not synchronized. At the end of the job $IMEDIR is automatically removed. This example was tested on Owens.

      #!/bin/bash
      # Serial job that uses $IMEDIR for temporary files (read/write)
      #SBATCH --job-name=temporary_to_ime
      #SBATCH --nodes=1 --ntasks=28 --gres=ime
      #SBATCH --time=1:00:00
      #SBATCH --account={MYACCT} # put your primery account group, like PAS1234
      module load bwa
      
      # Change to job-specific IME temporary directory
      # $IMEDIR is automatically deleted at end of job
      cp upstream1000.fa $IMEDIR
      cd $IMEDIR
      
      # Run program, assuming temporary files will be written to current directory
      bwa index upstream1000.fa
      
      # check status
      ls -l
      ime-ctl -s $IMEDIR/upstream1000.fa.bwt
      # Note: No need to purge temporary files because directory will be deleted at end of job
      

      ime-ctl -s will display the file's status, for example

      File: `/ime/scratch/{your file location}/4239XXX.owens-batch.ten.osc.edu/upstream1000.fa.bwt'
      Number of bytes:
      Dirty: 43886080
      Clean: 0
      Syncing: 0
      

      As you see, the file is completely "Dirty", so you know that it is only on IME side.

      Output files to be kept

      In this example an output file is written to IME. At the end of the job the file is synchronized and then purged from IME, remaining on the scratch file system.

      #!/bin/bash
      # Serial job that writes output files to IME
      #SBATCH --job-name=output_to_ime
      #SBATCH --nodes=1 --ntasks=28 --gres=ime
      #SBATCH --time=1:00:00 
      #SBATCH --account={MYACCT} # put your primery account group, like PAS1234
      
      module load bwa
      
      # create working directory under IME, and copy input files.
      # The input files can be read from regular file system as well.
      export IME_WORKDIR=/ime/scratch/{your file location}
      mkdir $IME_WORKDIR
      cp upstream1000.fa $IME_WORKDIR
      cd $IME_WORKDIR
      
      # Run program, assuming output will be written to current directory (IME directory)
      bwa index upstream1000.fa
      
      # Wait for synchronization to complete (blocking)
      ime-ctl -b -R -r $IME_WORKDIR
      
      # checking status
      ls -l
      ime-ctl -s $IME_WORKDIR/upstream1000.fa.bwt
      

      Large read-only files used by multiple jobs

      Some workflows involve large input files that are used by many jobs but never modified. This example keeps the input file resident in IME, re-importing it as necessary. If the file is ever changed it must be manually purged or completely reloaded; this is not part of the job workflow.

      #!/bin/bash
      #Job with large read-only input file used by multiple jobs
      #SBATCH --job-name=read_only_to_ime
      #SBATCH --nodes=1 --ntasks=28 --gres=ime
      #SBATCH --time=1:00:00
      #SBATCH --account={MYACCT} # put your primery account group, like PAS1234
      
      module load bwa
      
      # We assume the input file is located in /fs/scratch, then we import to IME
      export IME_WORKDIR=/ime/scratch/{your file location}
      export INPUTFILE=/fs/scratch/{your file location}/upstream1000.fa
      
      # Get IME FUSE path for input file (changes /fs to /ime)
      export IME_INPUTFILE=$(echo $INPUTFILE | sed 's/^\/fs/\/ime/')
      
      # import to IME
      ime-ctl -b -i $IME_INPUTFILE
      
      cd $IME_WORKDIR
      
      # Run program
      bwa index upstream1000.fa
      
      # check status
      ls -l
      ime-ctl -s $IME_WORKDIR/upstream1000.fa.bwt
      

      Checkpoint files

      Checkpoint files are written by a program to allow restart in case the program is terminated before completion. If the program completes the checkpoint file is discarded. Checkpoint files should not be written to $IMEDIR because they need to persist beyond the end of the job. In this example the checkpoint files are left in IME if the script does not complete and must be manually recovered or purged, or used directly from IME by a subsequent job.

      #!/bin/bash
      # Job that writes checkpoint files to IME
      #SBATCH --job-name=checkpoint_to_ime
      #SBATCH --nodes=1 --ntasks=28 --gres=ime
      #SBATCH --time=1:00:00
      #SBATCH --account={MYACCT} # put your primery account group, like PAS1234
      
      module load qchem/5.1.1-openmp
      module list
      
      export CKPTDIR=/fs/scratch/{your file location}/ckptdir
      mkdir $CKPTDIR
      
      # Get IME FUSE path for checkpoint directory (changes /fs to /ime)
      export IME_CKPTDIR=$(echo $CKPTDIR | sed 's/^\/fs/\/ime/')
      
      # Set checkpoint path in Q-Chem
      export QCSCRATCH=$IME_CKPTDIR
      
      # Run program, writing checkpoint files to $IME_CKPTDIR
      qchem -save -nt $SLURM_NTASKS HF_water.in HF_water.out HF_water
      
      # If program completed successfully delete checkpoint files
      retVal=$?
      if [ $retVal -eq 0 ]; then
          rm -r $CKPTDIR
      fi
      exit $retVal
      
      # Note: If program did not complete successfully or job was killed, checkpoint
      #   files will remain in IME and can be synchronized and purged manually.
      #   Or they can be used directly from IME by a subsequent job.
      

       

      Supercomputer: 
      Service: 

      HOWTO: Identify users on a project account and check status

      An eligible principal investigator (PI) heads a project account and can authorize/remove user accounts under the project account (please check our Allocations and Accounts documentation for more details). This document shows you how to identify users on a project account and check the status of each user. 

      Identify Users on a Project Account

      If you know the project acccount

      If the project account (projectID) is known, the OSCgetent command will list all users on the project:

      $ OSCgetent group projectID
      

      The returned information is in the format of:

      projectID:*:gid: list of user IDs
      

      gid is the group identifier number unique for the project account projectID. 

      For example, the command  OSCgetent group PZS0712 lists all users on the project account PZS0712 as below:

      $ OSCgetent group PZS0712
      PZS0712:*:5513:amarcum,guilfoos,hhamblin,kcahill,xwang
      

      Multiple groups can also be queried at once.

      For Example, the command OSCgetent group PZS0712 PZS0726 lists all users on both PZS0712 and PZS0726:

      PZS0712:*:5513:amarcum,guilfoos,hhamblin,kcahill,xwang
      PZS0726:*:6129:amarcum,kkappel
      

      Details on a project can also be obtained along with the user list using the OSCfinger command.

      $ OSCfinger -g projectID 
      

      This returns:

      Group: projectID                                  GID: XXXX
      Status: 'active/restricted/etc'                   Type: XX
      Principal Investigator: 'PI email'                Admins: NA
      Members: 'list of users'
      Category: NA
      Institution: 'affliated institution'
      Description: 'short description'
      ---
      

      If you don't know the project acccount, but know the username

      If the project account is not known, but the username is known, use the  OSCfinger command to list all of the groups the user belongs to:

      OSCfinger username
      

      The returned information is in the format of:

      Login: username                                   Name: First Last
      Directory: home directory path                    Shell: /bin/bash
      E-mail: user's email address
      Primary Group: user's primary project
      Groups: list of projects and other groups user is in
      Password Changed: date password was last changed  Password Expires: date password expires
      Login Disabled: TRUE/FALSE                             Password Expired: TRUE/FALSE
      Current Logins:
      Displays if user is currently logged in and from where/when
      

      For example, with the username as amarcum, the command OSCfinger amarcum returns the information as below:

      $ OSCfinger amarcum
      Login: amarcum                                    Name: Antonio Marcum
      Directory: /users/PZS0712/amarcum                 Shell: /bin/bash
      E-mail: amarcum@osc.edu
      Primary Group: PZS0712
      Groups: sts,ruby,l2supprt,oscall,clntstf,oscstaff,clntall,PZS0712,PZS0726
      Password Changed: May 12 2019 15:47 (calculated)  Password Expires: Aug 11 2019 12:05 AM
      Login Disabled: FALSE                             Password Expired: FALSE
      Current Logins:
      On since Mar 07 2019 12:12 on pts/14 from pitzer-login01.hpc.osc.edu
      ----
      

      If you don't know either the project account or user account

      If the project account or username is not known, use the OSCfinger -e   command with the '-e' flag to get the user account based on the user's name.

      Use the following command to list all of the user accounts associated with a First and Last name:

      $ OSCfinger -e 'First Last'
      

      For example, with user's first name as Summer and last name as Wang, the command

      OSCfinger -e 'Summer Wang' returns the information as below:

      $ OSCfinger -e 'Summer Wang'
      Login: xwang                                      Name: Summer Wang
      Directory: /users/oscgen/xwang                    Shell: /bin/bash
      E-mail: xwang@osc.edu
      Primary Group: PZS0712
      Groups: amber,abaqus,GaussC,comsol,foampro,sts,awsmdev,awesim,ruby,matlab,aasheats,mars,ansysflu,wrigley,lgfuel,l2supprt,fsl,oscall,clntstf,oscstaff,singadm,clntall,dhgremot,fsurfer,PZS0530,PCON0003,PZS0680,PMIU0149,PZS0712,PAS1448
      Password Changed: Jan 08 2019 11:41               Password Expires: Jul 08 2019 12:05 AM
      Login Disabled: FALSE                             Password Expired: FALSE
      ---
      

      Once you know the user account username, follow the discussions in the previous session to get all user accounts on the project. Please contact OSC Help if you have any questions. 

      Check the Status of a User

      Use the OSCfinger command to check the status of a user account as below:

      OSCfinger username
      

      For example, if the username is xwang, the command  OSCfinger xwang will return:

      $ OSCfinger xwang
      Login: xwang                                      Name: Summer Wang
      Directory: /users/oscgen/xwang                    Shell: /bin/bash
      E-mail: xwang@osc.edu
      Primary Group: PZS0712
      Groups: amber,abaqus,GaussC,comsol,foampro,sts,awsmdev,awesim,ruby,matlab,aasheats,mars,ansysflu,wrigley,lgfuel,l2supprt,fsl,oscall,clntstf,oscstaff,singadm,clntall,dhgremot,fsurfer,PZS0530,PCON0003,PZS0680,PMIU0149,PZS0712,PAS1448
      Password Changed: Jan 08 2019 11:41               Password Expires: Jul 08 2019 12:05 AM
      Login Disabled: FALSE                             Password Expired: FALSE
      ---
      
      • The home directory of xwang is   Directory: /users/oscgen/xwang 
      • The shell of xwang is bash ( Shell: /bin/bash ). If the information is Shell:/access/denied , it means this user account has been either archived or restricted. Please contact OSC Help if you'd like to reactivate this user account.
      • xwang@osc.edu is the associated email with the user account xwang; that is, all OSC emails related to the account xwang will be sent to xwang@osc.edu ( Mail forwarded to xwang@osc.edu ). Please contact OSC Help if the email address associated with this user account has been changed to ensure important notifications/messages/reminders from OSC may be received in a timely manner.

      Check the Usage and Quota of a User's Home Directory/Project Space

      All users see their file system usage statistics when logging in, like so:

      As of 2018-01-25T04:02:23.749853 userid userID on /users/projectID used XGB of quota 500GB and Y files of quota 1000000 files
      

      The information is from the file /users/reporting/storage/quota/*_quota.txt , which is updated twice a day. Some users may see multiple lines associated with a username, as well as information on project space usage and quota, if there is one. The usage and quota of the home directory of a username is provided by the line including the file server your home directory is on (for more information, please visit Home Directories), while others (generated due to file copy) can be safely ignored. 

      You can check any user's home directory or a project's project space usage and quota by running:

      grep -h 'userID' OR 'projectID' /users/reporting/storage/quota/*_quota.txt
      
      

      Here is an example of project PZS0712:

      $ grep -h PZS0712 /users/reporting/storage/quota/*_quota.txt
      As of 2019-03-07T13:55:01.000000 project/group PZS0712 on /fs/project used 262 GiB of quota 2048 GiB and 166987 files of quota 200000 files
      As of 2019-03-07T13:55:01.000000 userid xwang on /fs/project/PZS0712 used 0 GiB of quota 0 GiB and 21 files of quota 0 files
      As of 2019-03-07T13:55:01.000000 userid dheisterberg on /fs/project/PZS0712 used 262 GiB of quota 0 GiB and 166961 files of quota 0 files
      As of 2019-03-07T13:55:01.000000 userid amarcum on /fs/project/PZS0712 used 0 GiB of quota 0 GiB and 2 files of quota 0 files
      As of 2019-03-07T13:55:01.000000 userid root on /fs/project/PZS0712 used 0 GiB of quota 0 GiB and 2 files of quota 0 files
      As of 2019-03-07T13:55:01.000000 userid guilfoos on /fs/project/PZS0712 used 0 GiB of quota 0 GiB and 1 files of quota 0 files
      As of 2019-03-07T13:51:23.000000 userid amarcum on /users/PZS0712 used 399.86 MiB of quota 500 GiB and 8710 files of quota 1000000 files
      

      Here is an example for username amarcum:

      $ grep -h amarcum /users/reporting/storage/quota/*_quota.txt
      As of 2019-03-07T13:55:01.000000 userid amarcum on /fs/project/PZS0712 used 0 GiB of quota 0 GiB and 2 files of quota 0 files
      As of 2019-03-07T13:56:39.000000 userid amarcum on /users/PZS0645 used 4.00 KiB of quota 500 GiB and 1 files of quota 1000000 files
      As of 2019-03-07T13:56:39.000000 userid amarcum on /users/PZS0712 used 399.86 MiB of quota 500 GiB and 8710 files of quota 1000000 files
      

      Check the RU Usage for Projects and Users

      The OSCusage commnad can provide detailed information about computational usage for a given project and user.

      See https://www.osc.edu/resources/getting_started/oscusage for more details.

      Supercomputer: 
      Service: 

      HOWTO: Install Free Academic Ansys Desktop License for COVID19

      Free Academic Ansys Desktop License (2020 R1 version)

      Ansys, Inc. is generously providing a temporary free trial license for home use as a COVID-19 response for OSC academic users. You can download the installer and license to your desktop following the instructions below. The license will expire on Dec 31, 2020. (The original expiry date was May 31, 2020, but it has been extended to Dec 31, 2020)

      The extended license until Dec 31, 2020 is available now!

      This license is for OSC academic users only. 

      Notice: This temporary license key is not to be distributed, downloaded or made accessible to any other individual. It may only be used by you, the recipient, solely in connection with your university class or authorized university research. This temporary license key is not to be distributed, downloaded or made accessible to anyone in China or Russia.​

      Download

      Please click here (login required) to download the temporary Ansys license file and Ansys 2020R1 installer for Windows and Linux. You will need an OSC account and password to access. You don't need to download all the binary files. You can select only files that are related to your study. 

      If you downloaded and used Ansys with the license expiring on May 31 or Sep 30, 2020, you can download the new license file from here (login required). Just replacing the license file would be enough to use Ansys until Dec 31, 2020.   

      You can find supported platform information from here.

      Installation

      Windows:

      1. Extract downloaded files
      2. Install software by running setup.exe from the extracted directory.
        • Select Install Ansys Products
        • Follow the instructions
        • For Enter License Server Specification, click on Skip this step and configure later, follow instructions in step 3.
      3. Setting the temporary license file (academic.lic)
        1. Locate the downloaded license file. By default, your browser will write the file to your Downloads directory. The directions assume you are leaving it in the Downloads directory (%USERPROFILE%\Downloads\academic.lic).
        2. Should you choose to save the file to a different directory, please make note of that new file location so you can replace the text in quotes in the setx commands below with the new file location

        3. Open a command prompt: press the Windows key + at the same time, enter cmd, and press OK.
        4. In the command prompt, run the following commands to set appropriate environment variables to point to the license file: 
          • setx ANSYSLMD_LICENSE_FILE "%USERPROFILE%\Downloads\academic.lic"
          • setx DYNARDO_LICENSE_FILE "%USERPROFILE%\Downloads\academic.lic"
          • setx ESTERELD_LICENSE_FILE "%USERPROFILE%\Downloads\academic.lic"

      Linux:

      1. Extract downloaded files. For example,
        tar xvf STRUCTURES_2020R1_LINX64.tar
        unzip ELECTRONICS_2020R1_LINX64.zip
        1. Install software by running ./INSTALL from the extracted directory.
          • Select Install Ansys Products
          • Follow the instructions
          • For Enter License Server Specification, click on Skip this step and configure later, follow instructions in step 3.
        2. Setting the temporary license file (academic.lic)
          1. Locate the downloaded license file. 
          2. Set up the environment variables for the license file: 
            1. For csh:
              setenv ANSYSLMD_LICENSE_FILE "{path to}/academic.lic"
              setenv DYNARDO_LICENSE_FILE "{path to}/academic.lic"
              setenv ESTERELD_LICENSE_FILE "{path to}/academic.lic"
            2. For bash:
              export ANSYSLMD_LICENSE_FILE="{path to}/academic.lic"
              export DYNARDO_LICENSE_FILE="{path to}/academic.lic"
              export ESTERELD_LICENSE_FILE="{path to}/academic.lic"

        Support

        If you have any questions, please send an email to OSC Help.

        HOWTO: Install Local R Packages

         

        This document is obsoleted and Please refer to here for the latest version.

        This document shows you the steps to install R packages locally without root access on OSC's clusters. 

        R comes with a single library  $R_HOME/library which contains the standard and recommended packages. This is usually in a system location. On Owens, it is   /usr/local/R/3.3.2/lib64/R/library . R also has a default value for a directory where users can install their own R packages. On the Owens cluster, it is ~/R/x86_64-unknown-linux-gnu-library/3.3 if the default R-3.3.2 module is loaded. This directory doesn't exist by default. The first time a user installs an R package, R will ask the user if s/he wants to use the default location and if yes, will create the directory.

        A Simple Example

        First you need to load the module for R:

        module load R
        

        On Owens, the default R module is version 3.3.2 .

        Then fire up an R session:

        R
        

        To install package lattice, use this command inside R:

        > install.packages("lattice", repos="http://cran.r-project.org")
        

        It gives a warning: 

        Warning in install.packages("lattice") :
        'lib = "/usr/local/R/3.3.2/lib64/R/library"' is not writable
        Would you like to create a personal library
        ~/R/x86_64-unknown-linux-gnu-library/3.3
        to install packages into?  (y/n) 
        

        Answer y , and it will create the directory and install the package there.

        And, if you have errors similar to

        /opt/intel/18.0.3/compilers_and_libraries_2018.3.222/linux/compiler/include/complex(310): error #308: member "std::complex::_M_value" (declared at line 1346 of "/apps/gnu/7.3.0/include/c++/7.3.0/complex") is inaccessible 
          return __x / __y._M_value; 

        please execute the following command, and try to install it again. 

        echo "CXXFLAGS = -diag-disable 308" >> ~/.R/Makevars  

        Setting the Local R Library Path

        If you want to use another location rather than the default location, for example, ~/local/R_libs/ ,  you need to create the directory first:

        mkdir ~/local/R_libs
        

        Then type the following command inside R:

        > install.packages("lattice", repos="http://cran.r-project.org", lib="~/local/R_libs/")
        

        It is a bit of burden having to type the long string of library path every time. To avoid doing that, you can create a file .Renviron in your home directory, and add the following line to the file:

         R_LIBS_USER=~/local/R_libs/ 
        

        Whenever R is started, the directory ~/local/R_libs/ is added to the list of places to look for R packages and so:

        > install.packages("lattice", repos="http://cran.r-project.org")
        

        will have the same effect as the previous install.packages() command. 

        To see the directories where R searches for libraries, use the command:

        >.libPaths();
        
        

        Setting The Repository

        When you install an R package, you are asked which repository R should use. To set the repository and avoid having to specify this at every package install, create a file .Rprofile in your home directory. This is the start up code for R. Add the following line to the file:

        cat(".Rprofile: Setting R repository:")
        repo = getOption("repos") 
        # set up the server from which you will download the package.
        repo["CRAN"] = "http://cran.case.edu" 
        options(repos = repo)
        rm(repo)
        

        Now you only need to do 

        > install.packages("lattice")
        

        That will download the package lattice from  http://cran.case.edu and install it in ~/local/R_libs  .

        Updating Packages

        > update.packages() inside an R session is the simplest way to ensure that all the packages in your local R library are up to date. It downloads the list of available packages and their current versions, compares it with those installed and offers to fetch and install any that have later versions on the repositories.

        Note that attempting to update centrally installed packages will fail. You will have to use install.packages() to install a newer copy in your home directory instead. For example, if you are installing Bioconductor, the installation routine will try to update a bunch of centrally installed packages. To clone those into your personal library, tell biocLite() not to update any of the packages, and copy the list from the output into your own install.packages() command. See the example below (the c function creates an array of strings).

        > install.packages(c('Rmpi', 'assertthat', 'backports'))

        Removing packages

        > remove.packages("lattice") inside an R session to remove package lattice . An even easier way is just to go into the directory ~/local/R_libs and remove the directory lattice from there.

         

        Installing Packages from GitHub

        Users can install R packages directly from Github using devtools package as follows

        module load R/3.4.2
        R
        

                   

        >library(devtools)
        >install_github("author/package")
        

         

        References

        Add-on packages in R installation guide (http://cran.r-project.org/doc/manuals/R-admin.pdf)

        Supercomputer: 
        Service: 

        HOWTO: Install a MATLAB toolbox

        If you need to use a MATLAB toolbox that is not provided through our installations. You can follow these instructions, and if you have any difficulties you can contact OSC Help for assistance.

        A reminder: It is your responsibility to verify that your use of software packages on OSC’s systems including any 3rd party toolboxes (whether installed by OSC staff or by yourself)  complies with the packages’ license terms.

        Gather your materials

        First, we recommend making a new directory within your home directory in order to keep everything organized. You can use the unix command to make a new directory: "mkdir"

        Now you can download the toolbox either to your desktop, and then upload it to OSC, or directly download it using the "wget" utility (if you know the URL for the file).

        Now you can extract the downloaded file.

        Adding the path

        There are two methods on how to add the MATLAB toolbox path.

        Method 1: Load up the Matlab GUI and click on "Set Path" and "Add folder"

        Method 2: Use the "addpath" fuction in your script. More information on the function can be found here: https://www.mathworks.com/help/matlab/ref/addpath.html

        Running the toolbox

        Please refer to the instructions given alongside the toolbox. They should contain instructions on how to run the toolbox.

        Supercomputer: 
        Service: 
        Technologies: 
        Fields of Science: 

        HOWTO: Install your own Perl modules

        While we provide a number of Perl modules, you may need a module we do not provide. If it is a commonly used module, or one that is particularly difficult to compile, you can contact OSC Help for assistance, but we have provided an example below showing how to build and install your own Perl modules. Note, these instructions use "bash" shell syntax; this is our default shell, but if you are using something else (csh, tcsh, etc), some of the syntax may be different.

        CPAN Minus

        CPAN, the Comprehensive Perl Achive Network, is the primary source for publishing and fetching the latest modules and libraries for the Perl programming language. The default method for installing Perl modules using the "CPAN Shell", provides users with a great deal of power and flexibility but at the cost of a complex configuration and inelegant default setup.

         

        Setting Up CPAN Minus

        To use CPAN Minus, we must first load it, if it hasn't already been loaded. Note that this is not necessary if you loaded a version of Perl with the module load command.

        module load cpanminus
        

        Next, in order to use cpanminus, you will need to run the following command only ONCE:

        perl -I $CPANMINUS_INC -Mlocal::lib
        

         

        Using CPAN Minus

        In most cases, using CPAN Minus to install modules is as simple as issuing a command in the following form:

        cpanm [Module::Name]
        

        For example, below are three examples of installing perl modules:

        cpanm Math::CDF
        cpanm SET::IntervalTree
        cpanm DB_File
        

         

        Testing Perl Modules

        To test a perl module import, here are some examples below:

        perl -e "require Math::CDF"
        perl -e "require Set::IntervallTree"
        perl -e "require DB_File"
        

        The modules are installed correctly if no output is printed.

        What Local Modules are Installed in my Account?

        To show the local modules you have installed in your user account:

        perldoc perllocal
        

        Reseting Module Collection

        If you should ever want to start over with your perl module collection, delete the following folders:

        rm -r ~/perl5 
        rm -r ~/.cpanm
        

         

        Supercomputer: 
        Service: 

        HOWTO: Locally Installing Software

        Sometimes the best way to get access to a piece of software on the HPC systems is to install it yourself as a "local install". This document will walk you through the OSC-recommended procedure for maintaining local installs in your home directory or project space. This majority of this document describes the process of "manually" building and installing your software. We also show a partially automated approach through the use of a bash script in the Install Script section below.

        NOTE: Throughout this document we'll assume you're installing into your home directory, but you can follow the steps below in any directory for which you have read/write permissions.
        This document assumes you are familiar with the process of building software using "configure" or via editing makefiles, and only provides best practices for installing in your home directory.

        Getting Started

        Before installing your software, you should first prepare a place for it to live. We recommend the following directory structure, which you should create in the top-level of your home directory:

            local
            |-- src
            |-- share
                `-- lmodfiles
        

        This structure is how OSC organizes the software we provide. Each directory serves a specific purpose:

        • local - Gathers all the files related to your local installs into one directory, rather than cluttering your home directory. Applications will be installed into this directory with the format "appname/version". This allows you to easily store multiple versions of a particular software install if necessary.
        • local/src - Stores the installers -- generally source directories -- for your software. Also, stores the compressed archives ("tarballs") of your installers; useful if you want to reinstall later using different build options.
        • local/share/lmodfiles - The standard place to store module files, which will allow you to dynamically add or remove locally installed applications from your environment.

        You can create this structure with one command.

        NOTE: Ensure $HOME is the full path of your home directory. You can identify this from the command line with the command echo $HOME.

        After navigating to where you want to create the directory structure, run:

            mkdir -p $HOME/local/src $HOME/local/share/lmodfiles
        

        Installing Software

        Now that you have your directory structure created, you can install your software. For demonstration purposes, we will install a local copy of Git.

        First, we need to get the source code onto the HPC filesystem. The easiest thing to do is find a download link, copy it, and use the wget tool to download it on the HPC. We'll download this into $HOME/local/src:

            cd $HOME/local/src
            wget https://github.com/git/git/archive/v2.9.0.tar.gz
        

        Now extract the tar file:

            tar zxvf v2.9.0.tar.gz
        

        Next, we'll go into the source directory and build the program. Consult your application's documentation to determine how to install into $HOME/local/"software_name"/"version". Replace "software_name" with the software's name and "version" with the version you are installing, as demonstrated below. In this case, we'll use the configure tool's --prefix option to specify the install location.

        You'll also want to specify a few variables to help make your application more compatible with our systems. We recommend specifying that you wish to use the Intel compilers and that you want to link the Intel libraries statically. This will prevent you from having to have the Intel module loaded in order to use your program. To accomplish this, add CC=icc CFLAGS=-static-intel to the end of your invocation of configure. If your application does not use configure, you can generally still set these variables somewhere in its Makefile or build script.

        Then, we can build Git using the following commands:

            cd git-2.9.0
            autoconf # this creates the configure file
            ./configure --prefix=$HOME/local/git/2.9.0 CC=icc CFLAGS=-static-intel
            make && make install
        

        Your application should now be fully installed. However, before you can use it you will need to add the installation's directories to your path. To do this, you will need to create a module.

        Creating a Module

        Modules allow you to dynamically alter your environment to define environment variables and bring executables, libraries, and other features into your shell's search paths.

        We will be using the filename 2.9.0.lua ("version".lua). A simple Lua module for our Git installation would be:

        -- Local Variables
        local name = "git"
        local version = "2.9.0"
        
        -- Locate Home Directory
        local homedir = os.getenv("HOME")
        local root = pathJoin(homedir, "local", name, version)
        
        -- Set Basic Paths
        prepend_path("PATH", pathJoin(root, "bin"))
        prepend_path("LD_LIBRARY_PATH", root .. "/lib")
        prepend_path("LIBRARY_PATH", root .. "/lib")
        prepend_path("INCLUDE", root .. "/include")
        prepend_path("CPATH", root .. "/include")
        prepend_path("PKG_CONFIG_PATH", root .. "/lib/pkgconfig")
        prepend_path("MANPATH", root .. "/share/man")
        

        NOTE: For future module files, copy our sample modulefile from ~support/doc/modules/sample_module.lua. This module file follows the recommended design patterns laid out above and includes samples of many common module operations

        Our clusters use a Lua based module system. However, there is another module system based in TCL that will not be discussed in this HOWTO.                                                                         NOTE: TCL is cross-compatible and is converted to Lua when loaded. More documentation is available at https://www.tacc.utexas.edu/research-development/tacc-projects/lmod/ or by executing module help.

        Initializing Modules

        Any module file you create should be saved into your local lmodfiles directory ($HOME/local/share/lmodfiles). To prepare for future software installations, create a subdirectory within lmodfiles named after your software and add one module file to that directory for each version of the software installed.

        In the case of our Git example, you should create the directory $HOME/local/share/lmodfiles/git and create a module file within that directory named 2.9.0.lua.

        To make this module usable, you need to tell lmod where to look for it. You can do this by issuing the command module use $HOME/local/share/lmodfiles in our example. You can see this change by performing module avail. This will allow you to load your software using either module load git or module load git/2.9.0

        NOTE: module use$HOME/local/share/lmodfiles and module load "software_name" needs to be entered into the command line every time you enter a new session on the system.

        If you install another version later on (lets say version 2.9.1) and want to created a module file for it, you need to make sure you call it 2.9.1.lua. When loading Git, lmod will automatically load the newer version. If you need to go back to an older version, you can do so by specifying the version you want: module load git/2.9.0.

        To make sure you have the correct module file loaded, type which git which should give the output "~/local/git/2.9.0/bin/git" (NOTE: ~ is equivalent to $HOME). 

        To make sure the software was installed correctly and that the module is working, type git --version which shuold give the output "git version 2.9.0".

        Automating With Install Script

        Simplified versions of the scripts used to manage the central OSC software installations are provided at ~support/share/install-script. The idea is that you provide the minimal commands needed to obtain, compile, and install the software (usually some variation on wget, tar, ./configure, make, and make install) in a script, which then sources an OSC-maintained template that provides all of the "boilerplate" commands to create and manage a directory structure similar to that outlined in the Getting Started section above. You can copy an example install script from ~support/share/install-script/install-osc_sample.sh and follow the notes in that script, as well as in ~support/share/install-script/README.md, to modify it to install software of your choosing.

        NOTE: By default, the install script puts the module files in $HOME/osc_apps/lmodfiles, so you will need to run module use $HOME/osc_apps/lmodfiles and module load [software-name] every time you enter a new session on the system and want to use the software that you have installed.

        Further Reading

        For more information about modules, be sure to read the webpage indicated at the end of module help. If you have any questions about modules or local installations, feel free to contact the OSC Help Desk and oschelp@osc.edu.

        Supercomputer: 
        Service: 

        HOWTO: Manage Access Control List (ACLs)

        An ACL (access control list) is a list of permissions associated with a file or directory. These permissions allow you to restrict access to a certain file or directory by user or group. 

        OSC supports NFSv4 ACL on our home directory and POSIX ACL on our project and scratch file systems. Please see the how to use NFSv4 ACL for home directory ACL management and how to use POSIX ACL for managing ACLs in project and scratch file systems. 

        Supercomputer: 
        Service: 

        HOWTO: Use NFSv4 ACL

        This document shows you how to use the NFSv4 ACL permissions system. An ACL (access control list) is a list of permissions associated with a file or directory. These permissions allow you to restrict access to a certian file or directory by user or group. NFSv4 ACLs provide more specific options than typical POSIX read/write/execute permissions used in most systems.

        These commands are useful for managing ACLs in the dir locations of /users/<project-code>.

        Understanding NFSv4 ACL

        This is an example of an NFSv4 ACL

        A::user@nfsdomain.org:rxtncy
        
        A::alice@nfsdomain.org:rxtncy
        
        A::alice@nfsdomain.org:rxtncy
        
        A::alice@nfsdomain.org:rxtncy
        

        The following sections will break down this example from left to right and provide more usage options

         

        ACE Type

        The 'A' in the example is known as the ACE (access control entry) type. The 'A' denotes "Allow" meaning this ACL is allowing the user or group to perform actions requiring permissions. Anything that is not explicitly allowed is denied by default.

        Note: 'D' can denote a Deny ACE. While this is a valid option, this ACE type is not reccomended since any permission that is not explicity granted is automatically denied meaning Deny ACE's can be redundant and complicated.

         

        ACE Flags

        The above example could have a distinction known as a flag shown below

        A:d:user@osc.edu:rxtncy
        

        The 'd' used above is called an inheritence flag. This makes it so the ACL set on this directory will be automatically established on any new subdirectories. Inheritence flags only work on directories and not files. Multiple inheritence flags can be used in combonation or omitted entirely. Examples of inheritence flags are listed below:

        Flag Name Function
        d directory-inherit New subdirectories will have the same ACE
        f file-inherit New files will have the same ACE minus the inheritence flags 
        n no-propogate inherit New subdirectories will inherit the ACE minus the inheritence flags
        i inherit-only New files and subdirectories will have this ACE but the ACE for the directory with the flag is null

         

        ACE Principal

        The 'user@nfsdomain.org' is a principal. The principle denotes the people the ACL is allowing access to. Principals can be the following:

        • A named user
        • Special principals
          • OWNER@
          • GROUP@
          • EVERYONE@
        • A group
          • Note: When the principal is a group, you need to add a group flag, 'g', as shown in the below example
          • A:g:group@osc.edu:rxtncy
            

         

        ACE Permissions

        The 'rxtncy' are the permissions the ACE is allowing. Permissions can be used in combonation with each other. A list of permissions and what they do can be found below:

        Permission Function
        r read-data (files) / list-directory (directories)
        w write-data (files) / create-file (directories)
        a append-data (files) / create-subdirectory (directories)
        x execute (files) / change-directory (directories)
        d delete the file/directory
        D delete-child : remove a file or subdirectory from the given directory (directories only)
        t read the attributes of the file/directory
        T write the attribute of the file/directory
        n read the named attributes of the file/directory
        N write the named attributes of the file/directory
        c read the file/directory ACL
        C write the file/directory ACL
        o change ownership of the file/directory

         

        Note: Aliases such as 'R', 'W', and 'X' can be used as permissions. These work simlarly to POSIX Read/Write/Execute. More detail can be found below.

        Alias Name Expansion
        R Read rntcy
        W Write watTNcCy (with D added to directory ACE's
        X Execute xtcy

         

        Using NFSv4 ACL

        This section will show you how to set, modify, and view ACLs

         

        Set and Modify ACLs

        To set an ACE use this command:

        nfs4_setfacl [OPTIONS] COMMAND file
        

        To modify an ACE, use this command:

        nfs4_editfacl [OPTIONS] file
        

        Where file is the name of your file or directory. More information on Options and Commands can be found below.

         

        Commands

        Commands are only used when first setting an ACE. Commands and their uses are listed below.

        COMMAND FUNCTION
        -a acl_spec [index] add ACL entries in acl_spec at index (DEFAULT: 1)
        -x acl_spec | index remove ACL entries or entry-at-index from ACL
        -A file [index] read ACL entries to add from file
        -X file  read ACL entries to remove from file
        -s acl_spec set ACL to acl_spec (replaces existing ACL)
        -S file read ACL entries to set from file
        -m from_ace to_ace modify in-place: replace 'from_ace' with 'to_ace'

         

        Options

        Options can be used in combination or ommitted entirely. A list of options is shown below:

        OPTION NAME FUNCTION
        -R recursive Applies ACE to a directory's files and subdirectories
        -L logical Used with -R, follows symbolic links
        -P physical Used with -R, skips symbolic links
         

         

        View ACLs

        To view ACLs, use the following command:

        nfs4_getfacl file
        

        Where file is your file or directory

        Use cases

        Share data in your home directory with other users

        Assume that you want to share a directory (e.g data) and its files and subdirectories, but it is not readable by other users,

        > ls -ld /users/PAA1234/john/data
        drwxr-x--- 3 john PAA1234 4096 Nov 21 11:59 /users/PAA1234/john/data
        

        set an ACL to the directory 'data'  to allow a specific user for access:

        > cd /users/PAA1234/john
        > nfs4_setfacl -R -a A:df:userid@osc.edu:RX data
        

        or  to to allow a specific group for access:

        > cd /users/PAA1234/john
        > nfs4_setfacl -R -a A:dfg:groupname@osc.edu:RX data

        You can repeat the above commands to add more users or groups.

         

        Supercomputer: 
        Service: 

        HOWTO: Use POSIX ACL

        This document shows you how to use the POSIX ACL permissions system. An ACL (access control list) is a list of permissions associated with a file or directory. These permissions allow you to restrict access to a certian file or directory by user or group.

        These commands are useful for project and scratch dirs located in /fs/proejct, /fs/scratch, /fs/ess.

        Understanding POSIX ACL

        An example of a basic POSIX ACL would look like this:

        # file: foo.txt 
        # owner: tellison 
        # group: PZSXXXX 
        user::rw- 
        group::r-- 
        other::r--

        The first three lines list basic information about the file/directory in question: the file name, the primary owner/creator of the file, and the primary group that has permissions on the file. The following three lines show the file access permissions for the primary user, the primary group, and any other users. POSIX ACLs use the basic rwx permissions, explaned in the following table:

        Permission Explanation
        r Read-Only Permissions
        w Write-Only Permissions
        x

        Execute-Only Permissions

         

        Using POSIX ACL

        This section will show you how to set and view ACLs, using the setfacl and getfacl commands

        Viewing ACLs with getfacl

        The getfacl command displays a file or directory's ACL. This command is used as the following

        $ getfacl [OPTION] file

        Where file is the file or directory you are trying to view. Common options include:

        Flag Description
        -a/--access Display file access control list only
        -d/--default Display default access control list only (only primary access), which determines the default permissions of any files/directories created in this directory
        -R/--recursive Display ACLs for subdirectories
        -p/--absolute-names Don't strip leading '/' in pathnames

        Examples:

        A simple getfacl call would look like the following:

        $ getfacl foo.txt 
        # file: foo.txt
        # owner: user
        # group: PZSXXXX
        user::rw-
        group::r--
        other::r--

        A recursive getfacl call through subdirectories will list each subdirectories ACL separately

        $ getfacl -R foo/
        # file: foo/
        # owner: user
        # group: PZSXXXX
        user::rwx
        group::r-x
        other::r-x
        
        # file: foo//foo.txt
        # owner: user
        # group: PZSXXXX
        user::rwx
        group::---
        other::---
        
        # file: foo//bar
        # owner: user
        # group: PZSXXXX
        user::rwx
        group::---
        other::---
        
        # file: foo//bar/foobar.py
        # owner: user
        # group: PZSXXXX
        user::rwx
        group::---
        other::---
        

        Setting ACLs with setfacl

        The setfacl command allows you to set a file or directory's ACL. This command is used as the following

        $ setfacl [OPTION] COMMAND file

        Where file is the file or directory you are trying to modify. 

        Commands and Options

        setfacl takes several commands to modify a file or directory's ACL

        Command Function
        -m/--modify=acl

        modify the current ACL(s) of files. Use as the following

        setfacl -m u/g:user/group:r/w/x file

        -M/--modify-file=file

        read ACL entries to modify from a file. Use as the following

        setfaclt -M file_with_acl_permissions file_to_modify

        -x/--remove=acl

        remove entries from ACL(s) from files. Use as the following

        setfaclt -x u/g:user/group:r/w/x file

        -X/--remove-file=file

        read ACL entries to remove from a file. Use as the following

        setfaclt -X file_with_acl_permissions file_to_modify

        -b/--remove-all Remove all extended ACL permissions

        Common option flags for setfacl are as follows:

        Option Function
        -R/--recursive Recurse through subdirectories
        -d/--default Apply modifications to default ACLs
        --test test ACL modifications (ACLs are not modified

        Examples

        You can set a specific user's access priviledges using the following

        setfacl -m u:username:-wx foo.txt

        Similarly, a group's access priviledges can be set using the following

        setfacl -m g:PZSXXXX:rw- foo.txt

        You can remove a specific user's access using the following

        setfacl -x user:username foo.txt

        Grant a user recursive read access to a dir and all files/dirs under it (notice that the capital 'X' is used to provide execute permissions only to dirs and not files):

        setfacl -R -m u:username:r-X shared-dir

        Set a dir so that any newly created files or dirs under will inherit the parent dirs facl:

        setfacl -d -m u:username:r-X shared-dir

        HOWTO: Reduce Disk Space Usage

        This HOWTO will demonstrate how to lower ones' disk space usage. The following procedures can be applied to all of OSC's file systems.

        We recommend users regularly check their data usage and clean out old data that is no longer needed.

        Users who need assistance lowering their data usage can contact OSC Help.

        Preventing Excessive Data Usage Before It Starts

        Users should ensure that their jobs are written in such a way that temporary data is not saved to permanent file systems, such as the project space file system or their home directory.

        If your job copies data from the scratch file system or its node's local disk ($TMPDIR) back to a permanent file system, such as the project space file system or a home directory ( /users/PXX####/xxx####/), you should ensure you are only copying the files you will need later.

        Identifying Old and Large Data

        The following commands will help you identify old data using the find command.

        find commands may produce an excessive amount of output. To terminate the command while it is running, click CTRL + C.

        Find all files in a directory that have not been accessed in the past 100 days:

        This command will recursively search the users home directory and give a detailed listing of all files not accessed in the past 100 days.

        The last access time atime is updated when a file is opened by any operation, including grep, cat, head, sort, etc.

        find ~ -atime +100 -exec ls -l {} \;
        
        • To search a different directory replace ~ with the path you wish to search. A period . can be used to search the current directory.
        • To view files not accessed over a different time span, replace 100 with your desired number of days.
        • To view the total size in bytes of all the files found by find, you can add | awk '{s+=$5} END {print "Total SIZE (bytes): " s}' to the end of the command:
        find ~ -atime +100 -exec ls -l {} \;| awk '{s+=$5} END {print "Total SIZE (bytes): " s}'
        

        Find all files in a directory that have not been modified in the past 100 days:

        This command will recursively search the users home directory and give a detailed listing of all files not modified in the past 100 days.

        The last modified time mtime is updated when a file's contents are updated or saved. Viewing a file will not update the last modified time.

        find ~ -mtime +100 -exec ls -l {} \; 
        
        • To search a different directory replace ~ with the path you wish to search. A period . can be used to search the current directory.
        • To view files not modified over a different time span, replace 100 with your desired number of days.
        • To view the total size in bytes of all the files found by find, you can add | awk '{s+=$5} END {print "Total SIZE (bytes): " s}' to the end of the command:
        find ~ -mtime +100 -exec ls -l {} \;| awk '{s+=$5} END {print "Total SIZE (bytes): " s}'
        

        List files larger than a specified size:

        Adding the -size <size> option and argument to the find command allows you to only view files larger than a certain size. This option and argument can be added to any other find command.

        For example, to view all files in a users home directory that are larger than 1GB:

        find ~ -size 1G -exec ls -l {} \;
        

        Deleting Identified Data

        CAUTION: Be careful when deleting files. Be sure your command will do what you want before running it. Extra caution should be used when deleting files from a file system that is not backed up, such as the scratch file system.

        If you no longer need the old data, you can delete it using the rm command.

        If you need to delete a whole directory tree (a directory and all of its subcontents, including other directories), you can use the rm -R command.

        For example, the following command will delete the data directory in a users home directory:

        rm -R ~/data
        

        If you would like to be prompted for confirmation before deleting every file, use the -i option.

        rm -Ri ~/data 
        

        Enter y or n when prompted. Simply pressing the enter button will default to n.

        Deleting files found by find

        The rm command can be combined with any find command to delete the files found. The syntax for doing so is:

        find <location> <other find options> -exec rm -i {} \;
        

        Where <other find options> can include one or more of the options -atime <time>, -mtime <time>, and -size <size>.

        The following command would find all files in the ~/data directory 1G or larger that have not been accessed in the past 100 days, and then prompt for confirmation to delete each file:

        find ~/data -atime +100 -size 1G -exec rm -i {} \;
        

        If you are absolutely sure the files identified by find are okay to delete you can remove the -i option to rm and you will not be prompted. Extreme caution should be used when doing so!

        Archiving Data

        If you still need the data but do not plan on needing the data in the immediate future, contact OSC Help to discuss moving the data to an archive file system. Requests for data to be moved to the archive file system should be larger than 1TB.

        Compressing

        If you need the data but do not access the data frequently, you should compress the data using tar or gzip.

        Moving Data to a Local File System

        If you have the space available locally you can transfer your data there using sftp or Globus.

        Globus is recommended for large transfers.

        The OnDemand File application should not be used for transfers larger than 1GB.

        Supercomputer: 
        Service: 

        HOWTO: Submit Homework to Repository at OSC

        This page outlines a way a professor can set up a file submission system at OSC for his/her classroom project.

        Usage for Professor

        After connecting to OSC system, professor runs submit_prepare as

        $ /users/PZS0645/support/bin/submit_prepare

        Follow the instruction and provided the needed information (name of the assignment, TA username if appropriate, a size limit if not the default 1000MB per student, and whether or not you want the email notification of a submit). It will create a designated directory where students submit their assignments, as well as generate submit for students used to submit homework to OSC, both of which are located in the directory specified by the professor.

        Note:

        The PI can also enforce the deadline by simply changing the permission of the submission directory or renaming the submission directory at the deadline.

        (Only works on Owens): One way is to use at command following the steps below:

        • Use at command to specify the deadline:
        at [TIME]

        where TIME is formatted HH:MM AM/PM MM/DD/YY. For example:

        at 2:30 PM 08/21/2017
        • After running this command, run:
        $ chmod 700 [DIRECTORY]

        where DIRECTORY is the assignment folder to be closed off.

        • Enter [ctrl+D] to submit this command.

        The permission of DIRECTORY will be changed to 700 at 2:30PM, August 21, 2018. After that, the student will get an error message when he/she tries to submit an assignment to this directory.

        Usage for Students

        A student should create one directory which includes all the files he/she wants to submit before running this script to submit his/her assignment. Also, the previous submission of the same assignment from the student will be replaced by the new submission. 

        To submit the assignment, the student runs submit after connecting to OSC system as

        $ /path/to/directory/from/professor/submit

        Follow the instructions. It will allow students to submit an assignment to the designated directory specified by the professor and send a confirmation email, or return an error message.

        Supercomputer: 
        Service: 

        HOWTO: Submit multiple jobs using parameters

        Often users want to submit a large amount of jobs all at once with each using different parameters for each job.  These parameters could be anything, including the path of a data file or different input values for a program.  This how-to will show you how you can do this using a simple python script, a CSV file, and a template script.  You will need to adapt this advice for your own situation.

        Consider the following batch script:

        #!/bin/bash
        #SBATCH --ntasks-per-node=2
        #SBATCH --time=1:00:00
        #SBATCH --job-name=week42_data8
        
        # Copy input data to the nodes fast local disk
        cp ~/week42/data/source1/data8.in $TMPDIR
        
        cd $TMPDIR
        
        # Run the analysis
        full_analysis data8.in data8.out
        
        # Copy results to proper folder
        cp  data8.out ~/week42/results

        Lets say you need to submit 100 of these jobs on a weekly basis.  Each job uses a different data file as input.  You recieve data from two different sources, and thus your data is located within two different folders.  All of the jobs from one week need to store their results in a single weekly results folder.  The output file name is based upon the input file name.

        Creating a Template Script

        As you can see, this job follows a general template.  There are three main parameters that change in each job:

        1. The week 
          • Used as part of the job name
          • Used to find the proper data file to copy to the nodes local disk
          • Used to copy the results to the correct folder
        2. The data source
          • Used to find the proper data file to copy to the nodes local disk
        3. The data file's name
          • Used as part of the job name
          • Used to find the proper data file to copy to the nodes local disk
          • Used to specify both the input and output file to the program full_analysis
          • Used to copy the results to the correct folder

        If we replace these parameters with variables, prefixed by the dollar sign $and surrounded by curly braces { }, we get the following template script:

        Slurm does not support using variables in the #SBATCH section, so we need to set the job name in the submit command.
        #!/bin/bash
        #SBATCH --ntasks-per-node=2
        #SBATCH --time=1:00:00
        
        # Copy input data to the nodes fast local disk 
        cp ~/${WEEK}/data/${SOURCE}/${DATA}.out $TMPDIR
        cd $TMPDIR
        
        # Run the analysis 
        full_analysis ${DATA}.in ${DATA}.out
        
        # Copy results to proper folder
        cp  ${DATA}.out ~/${WEEK}/results
        

        Automating Job Submission

        We can now use the sbatch --exportoption to pass parameters to our template script.  The format for passing parameters is:

        sbatch --job-name=name --export=var_name=value[,var_name=value...]

        Submitting a 100 jobs using sbatch --export option manually does not make our task much easier than modifying and submitting each job one by one.  To complete our task we need to automate the submission of our jobs.  We will do this by using a python script that submits our jobs using parameters it reads from a CSV file.  

        Note that python was chosen for this task for its general ease of use and understandability -- if you feel more comfortable using another scritping language feel free to interpret/translate this python code for your own use.

        Here is the script that submits the jobs using the parameters:

        The below script needs python3 to run.
        Try using python3 submit_jobs.py to run the script.
        #!/usr/bin/python3
        ## file: submit_jobs.py
        import os
        import csv, subprocess
        
        home = os.path.expanduser('~')
        parameter_file_full_path = home + "/testDir/jobs_mult_params/week42/job_params.csv"
        
        with open(parameter_file_full_path, mode='r', newline='', encoding='utf-8-sig') as csvfile:
            reader = csv.reader(csvfile)
            for job in reader:
                submit_command = ("sbatch "
                    "--job-name={0}_{1} "
                    "--export=WEEK={0},SOURCE={1},DATA={2} template_1.sh").format(*job)
        
                print(submit_command)# Uncomment this line when done testing to use the submit command created
                # uncomment the following 3 lines when done testing to submit the jobs
        #        exit_status = subprocess.call(submit_command, shell=True)
        #        if exit_status is 1:  # Check to make sure the job submitted
        #            print("Job {0} failed to submit".format(submit_command))
        print("Done submitting jobs!")
        
        

        This script will open the CSV file specified by the variable parameter_file_full_path and step through the file line by line, submitting a job for each line using the lines values.  If the submit command returns a non-zero exit code, usually indicating it was not submitted, we will print this out to the display.  The jobs will be submitted using the general format:

        sbatch --export=WEEK=WEEK_VALUE,SOURCE=SOURCE_VALUE,DATA=DATA_VALUE template_1.sh

        Where WEEK_VALUE ,SOURCE_VALUE, and DATA_VALUE are the first, second, and third comma separated values in the CSV file's current row, and template_1.sh is the name of our template script.

        Creating a CSV File

        We now need to create a CSV file with parameters for each job.  This can be done with a regular text editor or using a spreadsheet editor such as excel.  By default you should use commas as your delimiter.  

        Here is our CSV file with parameters:

        week42,source1,data1
        week42,source1,data2
        week42,source1,data3
        ...
        week42,source2,data98
        week42,source2,data99
        week42,source2,data100
        

        The submit script would read in the first row of this CSV file and form and execute the command:

        sbatch --job-name=week42_source1 --export=WEEK=week42,SOURCE=source1,DATA=data1 template_1.sh

        Submitting Jobs

        Once all the above is done all you need to do to submit your jobs is make sure the CSV file is populated with the proper parameters and run the automatic submission script.  

        Before submitting a large amount of jobs for the first time using this method it is recommended you test our your implimentation with a small amount of jobs.

        HOWTO: Tune Performance

        Table of Contents

        Introduction

        Setup

        Debugging

        Hardware

        Performance Measurement

        Timing

        Profiling

        Help From the Compiler

        Compiler Optimization Reports

        Memory Optimizations

        Vectorization/Streaming

        OpenMP

        MPI

        GPU Accelerated Computing

        Summary

         

        Introduction

        This tutorial goes over techniques to tune the performance of your application. Keep in mind that correctness of results, code readability/maintainability, and portability to future systems are more important than performance. Some factors that can affect performance are:

        • Effective use of processor features for a high degree of internal concurrency in a single core
        • Mmory access patterns (memory access is slow compared to computation)
        • Use of an appropriate file system for file I/O
        • Scalability of algorithms
        • Compiler optimizations
        • Explicit parallelism

        We will be using code based on the HPCCD miniapp from Mantevo. It performs the Conjugate Gradient (CG) on a 3D chimney domain. CG is an iterative algorithm to numerically approximate the solution to a system of linear equations.

        Run code with:

        srun -n <numprocs> ./test_HPCCG nx ny nz
        

        where nx, ny, nz are the number of nodes in the x, y, and z dimension on each processor.

         

        Setup

        First start an interactive Pitzer Desktop session with OnDemand.

        You need to load intel 19.0.5 and mvapich2 2.3.3:

        module load intel/19.0.5 mvapich2/2.3.3
        

        Then clone the repository:

        git clone https://code.osu.edu/khuvis.1/performance2021_handson.git

         

        Debugging

        Debuggers let you execute your program one line at a time, inspect variable values, stop your programming at a particular line, and open a core file after the program crashes.

        For debugging, use the -g flag and remove optimzation or set to -O0. For example:

        icc -g -o mycode.c
        gcc -g -O0 -o mycode mycode.c

        To see compiler warnings and diagnostic options:

        icc -help diag
        man gcc
        

        ARM DDT

        ARM DDT is a commercial debugger produced by ARM. It can be loaded on all OSC clusters:

        module load arm-ddt
        

        To run a non-MPI program from the command line:

        ddt --offline --no-mpi ./mycode [args]
        

        To run an MPI program from the command line:

        ddt --offline -np num.procs ./mycode [args]
        Hands On

        Compile and run the code:

        make
        srun -n 2 ./test_HPCCG 150 150 150

        You should have received the following error message at the end of the program output:

        ===================================================================================
        =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
        =   PID 308893 RUNNING AT p0200
        =   EXIT CODE: 11
        =   CLEANING UP REMAINING PROCESSES
        =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
        ===================================================================================
        YOUR APPPLICATIN TERMINATED WITH EXIT STRING: Segmentation fault (signal 11)
        This typically referes to a problem with your application.
        Please see tthe FAQ page for debugging suggestions
        

        Set compiler flags -O0 -g  to CPP_OPT_FLAGS in Makefile. Then recompile and run with ARM DDT:

        make clean; make
        module load arm-ddt
        ddt -np 2 ./test_HPCCG 150 150 150
        

         

        Solution

        ddt_handson.png

        When DDT stops on the segmentation fault, the stack is in the YAML_Element::~YAML_Element function of YAML_Element.cpp. Looking at this function, we see that the loop stops at children.size() instead of children.size()-1. So, line 13 should be changed from

        for(size_t i=0; i<=children.size(); i++) {
        

        to

        for(size_t i=0; i<children.size(); i++) {
        

         

        Hardware

        On Pitzer, there are 40 cores per node (20 cores per socket and 2 sockets per node). There is support for AVX512, vector length 8 double or 16 single precision values and fused multiply-add. (There is hardware support for 4 thread per core, but it is currently not enabled on OSC systems.)

        There are three cache levels on Pitzer, and the statistics are shown in the table below:

        Pitzer Cache Statistics
        Cache level Size (KB) Latency (cycles) Max BW (bytes/cycle) Sustained BW (bytes/cycle)
        L1 DCU 32 4-6 192 133
        L2 MLC 1024 14 64 52
        L3 LLC 28160 50-70 16 15

        Never do heavy I/O in your home directory. Home directories are for long-term storage, not scratch files.

        One option for I/O intensive jobs is to use the local disk on a compute node. Stage files to and from your home directory into $TMPDIR using the pbsdcp command (e.g. pbsdcp file1 file2 $TMPDIR), and execute the program in $TMPDIR.

        Another option is to use the scratch file system ($PFSDIR). This is faster than other file systems, good for parallel jobs, and may be faster than local disk.

        For more information about OSC's file system, click here.

        For example batch scripts showing the use of $TMPDIR and $PFSDIR, click here.

        For more information about Pitzer, click here.

         

        Performance Measurement

        FLOPS stands for "floating point operations per second." Pitzer has a theoretical maximum of 720 teraflops. With the LINPACK benchmark of solving a dense system of linear equations, 543 teraflops. With the STREAM benchmark, which measures sustainable memory bandwidth and the corresponding computation rate for vector kernels, copy: 299095.01 MB/s, scale: 298741.01 MB/s, add: 331719.18 MB/s, and traid: 331712.19 MB/s. Application performance is typically much less than peak/sustained performance since applications usually do not take full advantage of all hardware features.

         

        Timing

        You can time a program using the /usr/bin/time command. It gives results for user time (CPU time spent running your program), system time (CPU time spent by your program in system calls), and elapsed time (wallclock). It also shows % CPU, which is (user + system) / elapsed, as well as memory, pagefault, swap, and I/O statistics.

        /usr/bin/time j3
        5415.03user 13.75system 1:30:29elapsed 99%CPU \
        (0avgtext+0avgdata 0maxresident)k \
        0inputs+0outputs (255major+509333minor)pagefaults 0 swaps
        

        You can also time portions of your code:

          C/C++ Fortran 77/90 MPI (C/C++/Fortran)
        Wallclock

        time(2), difftime(3),

        getrusage(2)

        SYSTEM_CLOCK(2) MPI_Wtime(3)
        CPU times(2) DTIME(3), ETIME(3) X

         

        Profiling

        A profiler can show you whether code is compute-bound, memory-bound, or communication bound. Also, it shows how well the code uses available resources and how much time is spent in different parts of your code. OSC has the following profiling tools: ARM Performance Reports, ARM MAP, Intel VTune, Intel Trace Analyzer and Collector (ITAC), Intel Advisor, TAU Commander, and HPCToolkit.

        For profiling, use the -g flag and specify the same optimization level that you normally would normally use with -On. For example:

        icc -g -O3 -o mycode mycode.c

        Look for

        • Hot spots (where most of the time is spent)
        • Excessive number of calls to short functions (use inlining!)
        • Memory usage (swapping and thrashing are not allowed at OSC)
        • % CPU (low CPU utilization may mean excessive I/O delays).

        ARM Performance Reports

        ARM PR works on precompiled binaries, so the -g flag is not needed. It gives a summary of your code's performance that you can view with a browser.

        For a non-MPI program:

        module load arm-pr
        perf-report --no-mpi ./mycode [args]
        

        For an MPI program:

        module load arm-pr
        perf-report --np num_procs ./mycode [args]

        ARM MAP

        Interpreting this profile requires some expertise. It gives details about your code's performance. You can view and explore the resulting profile using an ARM client.

        For a non-MPI program:

        module load arm-map
        map --no-mpi ./mycode [args]
        

        For an MPI program:

        module load arm-pr
        map --np num_procs ./mycode [args]

        For more information about ARM Tools, view OSC resources or visit ARM's website.

        Intel Trace Analyzer and Collector (ITAC)

        ITAC is a graphical tool for profiling MPI code (Intel MPI).

        To use:

        module load intelmpi # then compile (-g) code
        mpiexec -trace ./mycode
        

        View and explore the results using a GUI with traceanalyzer:

        traceanalyzer <mycode>.stf

         

        Help From the Compiler

        HPC software is traditionally written in Fortran or C/C++. OSC supports several compiler families. Intel (icc, icpc, ifort) usually gives fastest code on Intel architecture). Portland Group (PGI - pgcc, pgc++, pgf90) is good for GPU programming, OpenACC. GNU (gcc, g++, gfortran) is open source and universally available.

        Compiler options are easy to use and let you control aspects of the optimization. Keep in mind that different compilers have different values for options.  For all compilers, any highly optimized builds, such as those employing the options herein, should be thoroughly validated for correctness.

        Some examples of optimization include:

        • Function inlining (eliminating function calls)
        • Interprocedural optimization/analysis (ipo/ipa)
        • Loop transformations (unrolling, interchange, splitting, tiling)
        • Vectorization (operate on arrays of operands)
        • Automatic parallization of loops (very conservative multithreading)

        Compiler flags to try first are:

        • General optimization flags (-O2, -O3, -fast)
        • Fast math
        • Interprocedural optimization/analysis

        Faster operations are sometimes less accurate. For Intel compilers, fast math is default with -O2 and -O3. If you have a problem, use -fp-model precise. For GNU compilers, precise math is default with -O2 and -O3. If you want faster performance, use -ffast-math.

        Inlining is replacing a subroutine or function call with the actual body of the subprogram. It eliminates overhead of calling the subprogram and allows for more loop optimizations. Inlining for one source file is typically automatic with -O2 and -O3.

        Optimization Compiler Options

        Options for Intel compilers are shown below. Don't use -fast for MPI programs with Intel compilers. Use the same compiler command to link for -ipo with separate compilation. Many other optimization options can be found in the man pages. The recommended options are -O3 -xHost. An example is ifort -O3 program.f90.

        -fast Common optimizations
        -On

        Set optimization level (0, 1, 2, 3)

        -ipo Interprocedural optimization, multiple files
        -O3 Loop transforms
        -xHost Use highest instruction set available
        -parallel Loop auto-parallelization

        Options for PGI compilers are shown below. Use the same compiler command to link for -Mipa with separate compilation. Many other optimization options can be found in the man pages. The recommended option is -fast. An example is pgf90 -fast program.f90.

        -fast Common optimizations
        -On

        Set optimization level (0, 1, 2, 3, 4)

        -Mipa Interprocedural optimization
        -Mconcur Loop auto-parallelization

        Options for GNU compilers are shown below. Use the same compiler command to link for -Mipa with separate compilation. Many other optimization options can be found in the man pages. The recommended options are -O3 -ffast-math. An example is gfortran -O3 program.f90.

        -On Set optimization level (0, 1, 2, 3)
        N/A for separate compilation Interprocedural optimization
        -O3 Loop transforms
        -ffast-math Possibly unsafe floating point optimizations
        -march=native Use highest instruction set available

        Hands On

        Compile and run with different compiler options:

        time srun -n 2 ./test_HPCCG 150 150 150

        Using the optimal compiler flags, get an overview of the bottlenecks in the code with the ARM performance report:

        module load arm-pr
        perf-report -np 2 ./test_HPCCG 150 150 150
        
        Solution

         

        On Pitzer, sample times were:

        Compiler Option Runtime (seconds)
        -g 129
        -O0 -g 129
        -O1 -g 74
        -O2 -g 74
        -O3 -g

        74

        The performance report shows that the code is compute-bound.

        pr_handson.png

         

        Compiler Optimization Reports

        Compiler optimization reports let you understand how well the compiler is doing at optimizing your code and what parts of your code need work. They are generated at compile time and describe what optimizations were applied at various points in the source code. The report may tell you why optimizations could not be performed.

        For Intel compilers, -qopt-report and outputs to a file.

        For Portland Group compilers, -Minfo and outputs to stderr.

        For GNU compilers, -fopt-info and ouputs to stderr by default.

        A sample output is:

        LOOP BEGIN at laplace-good.f(10,7)
           remark #15542: loop was not vectorized: inner loop was already vectorized
        
           LOOP BEGIN at laplace-good.f(11,10)
           <Peeled loop for vectorization>
           LOOP END
        
           LOOP BEGIN at laplace-good.f(11,10)
              remark #15300: LOOP WAS VECTORIZED
           LOOP END
        
           LOOP BEGIN at laplace-good.f(11,10)
           <Remainder loop for vectorization>
              remark #15301: REMAINDER LOOP WAS VECTORIZED
           LOOP END
        
           LOOP BEGIN at laplace-good.f(11,10)
           <Remainder loop for vectorization>
           LOOP END
        LOOP END
        

        Hands On

        Add the compiler flag -qopt-report=5 and recompile to view an optimization report.

        Vectorization/Streaming

        Code is structured to operate on arrays of operands. Vector instructions are built into the processor. On Pitzer, the vector length is 16 single or 8 double precision. The following is a vectorizable loop:

        do i = 1,N
          a(i) = b(i) + x(1) * c(i)
        end do
        

        Some things that can inhibit vectorization are:

        • Loops being in the wrong order (usually fixed by compiler)
        • Loops over derived types
        • Function calls (can sometimes be fixed by inlining)
        • Too many conditionals
        • Indexed array accesses

        Hands On

        Use ARM MAP to identify the most expensive parts of the code.

        module load arm-map
        map -np 2 ./test_HPCCG 150 150 150
        

        Check the optimization report previously generated by the compiler (with -qopt-report=5) to see if any of the loops in the regions of the code are not being vectorized. Modify the code to enable vectorization and rerun the code.

        Solution

        vec_handson.pngMap shows that the most expensive segment of the code is lines 83-84 of HPC_sparsemv.cpp:

        for (int j=0; j< cur_nnz; j++)
          y[i] += cur_vals[j]*x[cur_inds[j]];
        

        The optimization report confirms that the loop was not vectorized due to a dependence on y.

        optreport_handson.png

        Incrementing a temporary variable instead of y[i], should enable vectorization:

        for (int j=0; j< cur_nnz; j++)
          sum += cur_vals[j]*x[cur_inds[j]];
        y[i] = sum;

        Recompiling and rerunning with change reduces runtime from 74 seconds to 63 seconds.​​

        Memory Optimizations

        Memory access is often the most important factor in your code's performance. Loops that work with arrays should use a stride of one whenever possible. C and C++ are row-major (store elements consecutively by row in 2D arrays), so the first array index should be the outermost loop and the last array index should be the innermost loop. Fortran is column-major, so the reverse is true. You can get factor of 3 or 4 speedup just by using unit stride. Avoid using arrays of derived data types, structs, or classes. For example, use structs of arrays instead of arrays of structures.

        Efficient cache usage is important. Cache lines are 8 words (64 bytes) of consecutive memory. The entire cache line is loaded when a piece of data is fetched.

        The code below is a good example. 2 cache lines are used for every 8 loop iterations, and it is unit stride:

        real*8 a(N), b(N)
        do i = 1,N
          a(i) = a(i) + b(i)
        end do
        
        ! 2 cache lines:
        ! a(1), a(2), a(3) ... a(8)
        ! b(1), b(2), b(3) ... b(8)
        

        The code below is a bad example. 1 cache line is loaded for each loop iteration, and it is not unit stride:

        TYPE :: node
          real*8 a, b, c, d, w, x, y, z
        END TYPE node
        TYPE(node) :: s(N)
        do i = 1, N
          s(i)%a = s(i)%a + s(i)%b
        end do
        
        ! cache line:
        ! a(1), b(1), c(1), d(1), w(1), x(1), y(1), z(1)
        

        Hands On

        Look again at the most expensive parts of the code using ARM MAP:

        module load arm-map
        map -np 2 ./test_HPCCG 150 150 150
        

        Look for any inefficient memory access patterns. Modify the code to improve memory access patterns and rerun the code. Do these changes improve performance?

        Solution

        Lines 110-148 of generate_matrix.cpp are nested loops:

        for (int ix=0; ix<nx; ix++) {
          for (int iy=0; iy<ny; iy++) {
            for (int iz=0; iz<nz; iz++) {
              int curlocalrow = iz*nx*ny+iy*nx+ix;
              int currow = start_row+iz*nx*ny+iy*nx+ix;
              int nnzrow = 0;
              (*A)->ptr_to_vals_in_row[curlocalrow] = curvalptr;
              (*A)->ptr_to_inds_in_row[curlocalrow] = curindptr;
              .
              .
              .
            }
          }
        }

        The arrays are accessed in a manner so that consecutive values of ix are accesssed in order. However, our loops are ordered so that the ix is the outer loop. We can reorder the loops so that ix is iterated in the inner loop:

        for (int iz=0; iz<nz; iz++) {
          for (int iy=0; iy<ny; iy++) {
            for (int ix=0; ix<nx; ix++) {
              .
              .
              .
            }
          }
        }

         This reduces the runtime from 63 seconds to 22 seconds.

        OpenMP

        OpenMP is a shared-memory, threaded parallel programming model. It is a portable standard with a set of compiler directives and a library of support functions. It is supported in compilers by Intel, Portland Group, GNU, and Cray.

        The following are parallel loop execution examples in Fortran and C. The inner loop vectorizes while the outer loop executes on multiple threads:

        PROGRAM omploop
        INTEGER, PARAMETER :: N = 1000
        INTEGER i, j
        REAL, DIMENSION(N, N) :: a, b, c, x
        ... ! Initialize arrays
        !$OMP PARALLEL DO
        do j = 1, N
          do i = 1, N
            a(i, j) = b(i, j) + x(i, j) * c(i, j)
          end do
        end do
        !$OMP END PARALLEL DO
        END PROGRAM omploop
        
        int main() {
          int N = 1000;
          float *a, *b, *c, *x;
        ... // Allocate and initialize arrays
        #pragma omp parallel for
          for (int i = 0; i < N; i++) {
            for (int j = 0; j < N; j++) {
              a[i*N+j] = b[i*N+j] + x[i*N+j] * c[i*N+j]
            }
          }
        }
        

        You can add an option to compile a program with OpenMP.

        For Intel compilers, add the -qopenmp option. For example, ifort -qopenmp ompex.f90 -o ompex.

        For GNU compilers, add the -fopenmp option. For example, gcc -fopenmp ompex.c -o ompex.

        For Portland group compilers, add the -mp option. For example, pgf90 -mp ompex.f90 -o ompex.

        To run an OpenMP program, requires multiple processors through SLURM (--N 1 -n 40) and set the OMP_NUM_THREADS environment variable (default is use all available cores). For the best performance, run at most one thread per core.

        An example script is:

        #!/bin/bash
        #SBATCH -J omploop
        #SBATCH -N 1
        #SBATCH -n 40
        #SBATCH -t 1:00
        
        export OMP_NUM_THREADS=40
        /usr/bin/time ./omploop
        

        For more information, visit http://www.openmp.org, OpenMP Application Program Interface, and self-paced turorials. OSC will host an XSEDE OpenMP workshop on November 5, 2019.

         

        MPI

        MPI stands for message passing interface for when multiple processes run on one or more nodes. MPI has functions for point-to-point communication (e.g. MPI_Send, MPI_Recv). It also provides a number of functions for typical collective communication patterns, including MPI_Bcast (broadcasts value from root process to all other processes), MPI_Reduce (reduces values on all processes to a single value on a root process), MPI_Allreduce (reduces value on all processes to a single value and distributes the result back to all processes), MPI_Gather (gathers together values from a group of processes to a root process), and MPI_Alltoall (sends data from all processes to all processes).

        A simple MPI program is:

        #include <mpi.h>
        #include <stdio.h>
        
        int main(int argc, char *argv[]) {
          int rank, size;
          MPI_INIT(&argc, &argv);
          MPI_Comm_rank(MPI_COMM_WORLD, &rank);
          MPI_COMM_size(MPI_COMM_WORLD, &size);
          printf("Hello from node %d of %d\n", rank size);
          MPI_Finalize();
          return(0);
        }
        

        MPI implementations available at OSC are mvapich2, Intel MPI (only for Intel compilers), and OpenMPI.

        MPI programs can be compiled with MPI compiler wrappers (mpicc, mpicxx, mpif90). They accept the same arguments as the compilers they wrap. For example, mpicc -o hello hello.c.

        MPI programs must run in batch only. Debugging runs may be done with interactive batch jobs. srun automatically determines exectuion nodes from PBS:

        #!/bin/bash
        #SBATCH -J mpi_hello
        #SBATCH -N 2
        #SBATCH --ntasks-per-node=40
        #SBATCH -t 1:00
        
        cd $PBS_O_WORKDIR
        srun ./hello
        

        For more information about MPI, visit MPI Forum and MPI: A Message-Passing Interface Standard. OSC will host an XSEDE MPI workshop on September 3-4, 2019. Self-paced tutorials are available here.

        Hands On

        Use ITAC to get a timeline of the run of the code.

        module load intelmpi
        LD_PRELOAD=libVT.so \
        mpiexec -trace -np 40 ./test_HPCCG 150 150 150
        traceanalyzer <stf_file>
        

        Look at the Event Timeline (under Charts). Do you see any communication patterns that could be replaced by a single MPI command?

        Solution

        Looking at the Event Timeline, we see that a large part of runtime is spent in the following communication pattern: MPI_Barrier, MPI_Send/MPI_Recv, MPI_Barrier. We also see that during this communication rank 0 is sending data to all other rank. We should be able to replace all of these MPI calls with a single call to MPI_Bcast.

        The relavent code is in lines 82-89 of ddot.cpp: 

          MPI_Barrier(MPI_COMM_WORLD);
          if(rank == 0) {
            for(int dst_rank=1; dst_rank < size; dst_rank++) {
              MPI_Send(&global_result, 1, MPI_DOUBLE, dst_rank, 1, MPI_COMM_WORLD);
            }
          }
          if(rank != 0) MPI_Recv(&global_result, 1, MPI_DOUBLE, 0, 1, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
          MPI_Barrier(MPI_COMM_WORLD);

        and can be replaced with:

        MPI_Bcast(&global_result, 1, MPI_DOUBLE, 0, MPI_COMM_WORLD);
        
        Supercomputer: 
        Service: 

        HOWTO: Tune VASP Memory Usage

        This article discusses memory tuning strategies for VASP.

        Data Distribution

        Typically the first approach for memory sensitive VASP issues is to tweak the data distribution (via NCORE or NPAR). The information at https://www.vasp.at/wiki/index.php/NPAR covers a variety of machines. OSC has fast communications via Infiniband. 

        Performance and memory consumption are dependent on the simulation model.  So we recommend a series of benchmarks varying the number of nodes and NCORE. The recommended initial value for NCORE is the ppn value used in the PBS directive. Of course, if this benchmarking is intractable then one must reexamine the model. For general points see:  https://www.vasp.at/wiki/index.php/Memory_requirements and https://www.vasp.at/wiki/index.php/Not_enough_memory  And of course one should start small and incrementally improve or scale up ones model.

        Rationalization

        Using the key parameters with respect to memory scaling listed at the VASP memory requirements page one can rationalize VASP memory usage.  The general approach is to study working calculations and then apply that understanding to scaled up or failing calculations.  This might help one identify if a calculation is close to a node's memory limit and happens to cross over the limit for reasons that might be out of ones control, in which case one might need to switch to higher memory nodes.

        Here is an example of rationalizing memory consumption.  Extract from a simulation output the key parameters:

        Dimension of arrays:
        k-points NKPTS = 18 k-points in BZ NKDIM = 18 number of bands NBANDS= 1344
        total plane-waves NPLWV = 752640
        ...
        dimension x,y,z NGXF= 160 NGYF= 168 NGZF= 224
        support grid NGXF= 320 NGYF= 336 NGZF= 448

        This yields 273 GB of memory according to
        https://www.vasp.at/wiki/index.php/Memory_requirements

        This estimate should be compared to actual memory reports.  See for example XDModD.  Note that most application software has an overhead in the ballpack of ten to twenty percent.  In addition, disk caching can consume significant memory.  Thus, one must adjust the memory estimate upwards.  It can then be comapred to the available memory per cluster and per cluster node type.

        Miscellaneous

        • In the INCAR input file NWRITE=3 is for verbose output and NWRITE=4 is for debugging output.
        • OSC does not have a VASP license and our staff has limited experience with it.  So investigate alternate forms of help:  ask within your research group and  post on the VASP mailing list.
        • Valgrind is a tool that can be used for many types of debugging including looking for memory corruptions and leaks. However, it slows down your code a very sizeable amount. This might not be feasible for HPC codes
        • ASAN (address sanitizer) is another tool that can be used for memory debugging. It is less featureful than Valgrind, but runs much quicker, and so will likely work with your HPC code.

         

        Supercomputer: 
        Service: 

        HOWTO: Use Address Sanitizer

        Address Sanitizer is a tool developed by Google detect memory access error such as use-after-free and memory leaks. It is built into GCC versions >= 4.8 and can be used on both C and C++ codes. Address Sanitizer uses runtime instrumentation to track memory allocations, which mean you must build your code with Address Sanitizer to take advantage of it's features.

        There is extensive documentation on the AddressSanitizer Github Wiki.

        Memory leaks can increase the total memory used by your program. It's important to properly free memory when it's no longer required. For small programs, loosing a few bytes here and there may not seem like a big deal. However, for long running programs that use gigabytes of memory, avoiding memory leaks becomes increasingly vital. If your program fails to free the memory it uses when it no longer needs it, it can run out of memory, resulting in early termination of the application. AddressSanitizer can help detect these memory leaks.

        Additionally, AddressSanitizer can detect use-after-free bugs. A use-after-free bug occurs when a program tries to read or write to memory that has already been freed. This is undefined behavior and can lead to corrupted data, incorrect results, and even program crashes.

        Building With Address Sanitzer

        We need to use gcc to build our code, so we'll load the gcc module:

        module load gnu/9.1.0

        The "-fsanitize=address" flag is used to tell the compiler to add AddressSanitizer.

        Additionally, due to some environmental configuration settings on OSC systems, we must also statically link against Asan. This is done using the "-static-libasan" flag.

        It's helpful to compile the code with debug symbols. AddressSanitizer will print line numbers if debug symbols are present. To do this, add the "-g" flag. Additionally, the "-fno-omit-frame-pointer" flag may be helpful if you find that your stack traces do not look quite correct.

        In one command, this looks like:

        gcc main.c -o main -fsanitize=address -static-libasan -g

        Or, splitting into separate compiling and linking stages:

        gcc -c main.c -fsanitize=address -g
        gcc main.o -o main -fsanitize=address -static-libasan

        Notice that both the compilation and linking steps require the "-fsanitize-address" flag, but only the linking step requires "-static-libasan". If your build system is more complex, it might make sense to put these flags in CFLAGS and LDFLAGS environment variables.

        And that's it!

        Examples

        No Leak

        First, let's look at a program that has no memory leaks (noleak.c):

        #include <stdio.h>
        #include <stdlib.h>
        #include <string.h>
        
        int main(int argc, const char *argv[]) {
            char *s = malloc(100);
            strcpy(s, "Hello world!");
            printf("string is: %s\n", s);
            free(s);
            return 0; 
        }

        To build this we run:

        gcc noleak.c -o noleak -fsanitize=address -static-libasan -g

        And, the output we get after running it:

        string is: Hello world!

        That looks correct! Since there are no memory leaks in this program, AddressSanitizer did not print anything. But, what happens if there are leaks?

        Missing free

        Let's look at the above program again, but this time, remove the free call (leak.c):

        #include <stdio.h>
        #include <stdlib.h>
        #include <string.h>
        
        int main(int argc, const char *argv[]) {
            char *s = malloc(100);
            strcpy(s, "Hello world!");
            printf("string is: %s\n", s);
            return 0;
        }

        Then, to build:

        gcc leak.c -o leak -fsanitize=address -static-libasan

        And the output:

        string is: Hello world!
        
        =================================================================
        ==235624==ERROR: LeakSanitizer: detected memory leaks
        
        Direct leak of 100 byte(s) in 1 object(s) allocated from:
            #0 0x4eaaa8 in __interceptor_malloc ../../.././libsanitizer/asan/asan_malloc_linux.cc:144
            #1 0x5283dd in main /users/PZS0710/edanish/test/asan/leak.c:6
            #2 0x2b0c29909544 in __libc_start_main (/lib64/libc.so.6+0x22544)
        
        SUMMARY: AddressSanitizer: 100 byte(s) leaked in 1 allocation(s).

        This is a leak report from AddressSanitizer. It detected that 100 bytes were allocated, but never freed. Looking at the stack trace that it provides, we can see that the memory was allocated on line 6 in leak.c

        Use After Free

        Say we found the above leak in our code, and we wanted to fix it. We need to add a call to free. But, what if we add it in the wrong spot?

        #include <stdio.h>
        #include <stdlib.h>
        #include <string.h>
        
        int main(int argc, const char *argv[]) {
            char *s = malloc(100);
            free(s);
            strcpy(s, "Hello world!");
            printf("string is: %s\n", s);
            return 0;
        }
        

        The above (uaf.c) is clearly wrong. Albiet a contrived example, the allocated memory, pointed to by "s", was written to and read from after it was freed.

        To Build:

        gcc uaf.c -o uaf -fsanitize=address -static-libasan

        Building it and running it, we get the following report from AddressSanitizer:

        =================================================================
        ==244157==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b0000000f0 at pc 0x00000047a560 bp 0x7ffcdf0d59f0 sp 0x7ffcdf0d51a0
        WRITE of size 13 at 0x60b0000000f0 thread T0
            #0 0x47a55f in __interceptor_memcpy ../../.././libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:790
            #1 0x528403 in main /users/PZS0710/edanish/test/asan/uaf.c:8
            #2 0x2b47dd204544 in __libc_start_main (/lib64/libc.so.6+0x22544)
            #3 0x405f5c  (/users/PZS0710/edanish/test/asan/uaf+0x405f5c)
        
        0x60b0000000f0 is located 0 bytes inside of 100-byte region [0x60b0000000f0,0x60b000000154)
        freed by thread T0 here:
            #0 0x4ea6f7 in __interceptor_free ../../.././libsanitizer/asan/asan_malloc_linux.cc:122
            #1 0x5283ed in main /users/PZS0710/edanish/test/asan/uaf.c:7
            #2 0x2b47dd204544 in __libc_start_main (/lib64/libc.so.6+0x22544)
        
        previously allocated by thread T0 here:
            #0 0x4eaaa8 in __interceptor_malloc ../../.././libsanitizer/asan/asan_malloc_linux.cc:144
            #1 0x5283dd in main /users/PZS0710/edanish/test/asan/uaf.c:6
            #2 0x2b47dd204544 in __libc_start_main (/lib64/libc.so.6+0x22544)
        
        SUMMARY: AddressSanitizer: heap-use-after-free ../../.././libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:790 in __interceptor_memcpy
        Shadow bytes around the buggy address:
          0x0c167fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
          0x0c167fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
          0x0c167fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
          0x0c167fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
          0x0c167fff8000: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
        =>0x0c167fff8010: fd fd fd fd fd fa fa fa fa fa fa fa fa fa[fd]fd
          0x0c167fff8020: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa
          0x0c167fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
          0x0c167fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
          0x0c167fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
          0x0c167fff8060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        Shadow byte legend (one shadow byte represents 8 application bytes):
          Addressable:           00
          Partially addressable: 01 02 03 04 05 06 07 
          Heap left redzone:       fa
          Freed heap region:       fd
          Stack left redzone:      f1
          Stack mid redzone:       f2
          Stack right redzone:     f3
          Stack after return:      f5
          Stack use after scope:   f8
          Global redzone:          f9
          Global init order:       f6
          Poisoned by user:        f7
          Container overflow:      fc
          Array cookie:            ac
          Intra object redzone:    bb
          ASan internal:           fe
          Left alloca redzone:     ca
          Right alloca redzone:    cb
          Shadow gap:              cc
        ==244157==ABORTING
        

        This is a bit intimidating. It looks like there's alot going on here, but it's not as bad as it looks. Starting at the top, we see what AddressSanitizer detected. In this case, a "WRITE" of 13 bytes (from our strcpy). Immediately below that, we get a stack trace of where the write occured. This tells us that the write occured on line 8 in uaf.c in the function called "main".

        Next, AddressSanitizer reports where the memory was located. We can ignore this for now, but depending on your use case, it could be helpful information.

        Two key pieces of information follow. AddressSanitizer tells us where the memory was freed (the "freed by thread T0 here" section), giving us another stack trace indicating the memory was freed on line 7. Then, it reports where it was originally allocated ("previously allocated by thread T0 here:"), line 6 in uaf.c.

        This is likely enough information to start to debug the issue. The rest of the report provides details about how the memory is laid out, and exactly which addresses were accessed/written to. You probably won't need to pay too much attention to this section. It's a bit "down in the weeds" for most use cases.

        Heap Overflow

        AddresssSanitizer can also detect heap overflows. Consider the following code (overflow.c):

        #include <stdio.h>
        #include <stdlib.h>
        #include <string.h>
        int main(int argc, const char *argv[]) {
            // whoops, forgot c strings are null-terminated
            // and not enough memory was allocated for the copy
            char *s = malloc(12);
            strcpy(s, "Hello world!");
            printf("string is: %s\n", s);
            free(s);
            return 0;
        }

        The "Hello world!" string is 13 characters long including the null terminator, but we've only allocated 12 bytes, so the strcpy above will overflow the buffer that was allocated. To build this:

        gcc overflow.c -o overflow -fsanitize=address -static-libasan -g -Wall

        Then, running it, we get the following report from AddressSanitizer:

        ==168232==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200000003c at pc 0x000000423454 bp 0x7ffdd58700e0 sp 0x7ffdd586f890
        WRITE of size 13 at 0x60200000003c thread T0
            #0 0x423453 in __interceptor_memcpy /apps_src/gnu/8.4.0/src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:737
            #1 0x5097c9 in main /users/PZS0710/edanish/test/asan/overflow.c:8
            #2 0x2ad93cbd7544 in __libc_start_main (/lib64/libc.so.6+0x22544)
            #3 0x405d7b  (/users/PZS0710/edanish/test/asan/overflow+0x405d7b)
        
        0x60200000003c is located 0 bytes to the right of 12-byte region [0x602000000030,0x60200000003c)
        allocated by thread T0 here:
            #0 0x4cd5d0 in __interceptor_malloc /apps_src/gnu/8.4.0/src/libsanitizer/asan/asan_malloc_linux.cc:86
            #1 0x5097af in main /users/PZS0710/edanish/test/asan/overflow.c:7
            #2 0x2ad93cbd7544 in __libc_start_main (/lib64/libc.so.6+0x22544)
        
        SUMMARY: AddressSanitizer: heap-buffer-overflow /apps_src/gnu/8.4.0/src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:737 in __interceptor_memcpy
        Shadow bytes around the buggy address:
          0x0c047fff7fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
          0x0c047fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
          0x0c047fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
          0x0c047fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
          0x0c047fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
        =>0x0c047fff8000: fa fa 00 fa fa fa 00[04]fa fa fa fa fa fa fa fa
          0x0c047fff8010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
          0x0c047fff8020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
          0x0c047fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
          0x0c047fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
          0x0c047fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        Shadow byte legend (one shadow byte represents 8 application bytes):
          Addressable:           00
          Partially addressable: 01 02 03 04 05 06 07 
          Heap left redzone:       fa
          Freed heap region:       fd
          Stack left redzone:      f1
          Stack mid redzone:       f2
          Stack right redzone:     f3
          Stack after return:      f5
          Stack use after scope:   f8
          Global redzone:          f9
          Global init order:       f6
          Poisoned by user:        f7
          Container overflow:      fc
          Array cookie:            ac
          Intra object redzone:    bb
          ASan internal:           fe
          Left alloca redzone:     ca
          Right alloca redzone:    cb
        ==168232==ABORTING

        This is similar to the use-after-free report we looked at above. It tells us that a heap buffer overflow occured, then goes on to report where the write happened and where the memory was originally allocated. Again, the rest of this report describes the layout of the heap, and probably isn't too important for your use case.

        C++ Delete Mismatch

        AddressSanitizer can be used on C++ codes as well. Consider the following (bad_delete.cxx):

        #include <iostream>
        #include <cstring>
        
        int main(int argc, const char *argv[]) {
            char *cstr = new char[100];
            strcpy(cstr, "Hello World");
            std::cout << cstr << std::endl;
        
            delete cstr;
            return 0;
        }

        What's the problem here? The memory pointed to by "cstr" was allocated with new[]. An array allocation must be deleted with the delete[] operator, not "delete".

        To build this code, just use g++ instead of gcc:

        g++ bad_delete.cxx -o bad_delete -fsanitize=address -static-libasan -g

        And running it, we get the following output:

        Hello World
        =================================================================
        ==257438==ERROR: AddressSanitizer: alloc-dealloc-mismatch (operator new [] vs operator delete) on 0x60b000000040
            #0 0x4d0a78 in operator delete(void*, unsigned long) /apps_src/gnu/8.4.0/src/libsanitizer/asan/asan_new_delete.cc:151
            #1 0x509ea8 in main /users/PZS0710/edanish/test/asan/bad_delete.cxx:9
            #2 0x2b8232878544 in __libc_start_main (/lib64/libc.so.6+0x22544)
            #3 0x40642b  (/users/PZS0710/edanish/test/asan/bad_delete+0x40642b)
        
        0x60b000000040 is located 0 bytes inside of 100-byte region [0x60b000000040,0x60b0000000a4)
        allocated by thread T0 here:
            #0 0x4cf840 in operator new[](unsigned long) /apps_src/gnu/8.4.0/src/libsanitizer/asan/asan_new_delete.cc:93
            #1 0x509e5f in main /users/PZS0710/edanish/test/asan/bad_delete.cxx:5
            #2 0x2b8232878544 in __libc_start_main (/lib64/libc.so.6+0x22544)
        
        SUMMARY: AddressSanitizer: alloc-dealloc-mismatch /apps_src/gnu/8.4.0/src/libsanitizer/asan/asan_new_delete.cc:151 in operator delete(void*, unsigned long)
        ==257438==HINT: if you don't care about these errors you may set ASAN_OPTIONS=alloc_dealloc_mismatch=0
        ==257438==ABORTING
        

        This is similar to the other AddressSanitizer outputs we've looked at. This time, it tells us there's a mismatch between new and delete. It prints a stack trace for where the delete occured (line 9) and also a stack trace for where to allocation occured (line 5).

        Performance

        The documentation states:

        This tool is very fast. The average slowdown of the instrumented program is ~2x

        AddressSanitizer is much faster than tools that do similar analysis such as valgrind. This allows for usage on HPC codes.

        However, if you find that AddressSanitizer is too slow for your code, there are compiler flags that can be used to disable it for specific functions. This way, you can use address sanitizer on cooler parts of your code, while manually auditing the hot paths.

        The compiler directive to skip analyzing functions is:

        __attribute__((no_sanitize_address)

         

        Supercomputer: 
        Technologies: 

        HOWTO: Use Cron and OSCusage for Regular Emailed Reports

        It is possible to utilize Cron and the OSCusage command to send regular usage reports via email

        Cron

        It is easy to create Cron jobs on the Owens and Pitzer clusters at OSC. Cron is a Linux utility which allows the user to schedule a command or script to run automatically at a specific date and time. A cron job is the task that is scheduled.

        Shell scripts run as a cron job are usually used to update and modify files or databases; however, they can perform other tasks, for example a cron job can send an email notification.

        Getting Help

        In order to use what cron has to offer, here is a list of the command name and options that can be used

        Usage: 
        crontab [options] file 
        crontab [options] 
        crontab -n [hostname] 
        Options: 
        -u  define user 
        -e edit user's crontab 
        -l list user's crontab 
        -r delete user's crontab 
        -i prompt before deleting 
        -n  set host in cluster to run users' crontabs 
        -c get host in cluster to run users' crontabs 
        -s selinux context 
        -x  enable debugging 

        Also, if this is your first time using cron, you will be asked to choose an editor for setting your cron job. Choose whatever you find to be easiest for you.

        Running a Cron Job

        To check for any running cron jobs on the server, use the command (As shown above)

        crontab -l 

        and to create and edit your cron job use the following command,

        crontab -e 

        Now, in order to write you first cron job, you need to be familiar with the formatting system that cron follows.

        Linux Crontab Format

        The formatting system has 6 fields, each field from 1-5 is used to define the date and time of the execution. The 6th field is used for the command or script to be executed. The format is the following,

        MIN HOUR DOM MON DOW CMD 

        where,

        figure 1: Cron’s formatting syntax

        Getting Notified by Email Using a Cron Job

        You can get an email notification using a cron job as mentioned earlier. The following is an example of a cron job that runs every minute and sends an email notification every minute,

        * * * * * {cmd} | mail -s "title of the email notification" {your email} 

        A user can also set up email notifications regarding usage by using the OSCusage cmd,

        12 15 * * * /opt/osc/bin/OSCusage | mail -s "OSC usage on $(date)" {your email} 2> /path/to/file/for/stdout/and/stderr 2>&1 

        This cron job will run every day at (15:12 or 3:12 PM).

        Using OSCusage

        The OSCusage command offers many options, the following is a list that pertains to that,

        $ /opt/osc/bin/OSCusage --help 
        usage: OSCusage.py [-h] [-u USER] 
        [-s {opt,pitzer,glenn,bale,oak,oakley,owens,ruby}] [-A] 
        [-P PROJECT] [-q] [-H] [-r] [-n] [-v] 
        [start_date] [end_date] 
        
        positional arguments: 
        start_date start date (default: 2020-04-23) 
        end_date end date (default: 2020-04-24) 
        
        optional arguments: 
        -h, --help show this help message and exit 
        -u USER, --user USER username to run as. Be sure to include -P or -A. (default: kalattar) 
        -s {opt,pitzer,glenn,bale,oak,oakley,owens,ruby}, --system {opt,pitzer,glenn,bale,oak,oakle 
        -A Show all 
        -P PROJECT, --project PROJECT project to query (default: PZS0715) 
        -q show user data 
        -H show hours 
        -r show raw 
        -n show job ID 
        -v do not summarize 

        As it can be seen, one could for example use OSCusage to receive information regarding another user’s usage with the -u option and write a cron script that is set up with email notification.

        Some other usage examples,

         OSCusage 2018-01-24 

        where the command specifies the usage’s start time. The end time could also be specified with,

        OSCusage 2018-01-24 2018-01-25 

        Terminating a Cron Job

        To terminate a cron job, you need to first determine the process id,

        ps aux | grep crontab 

        and then use,

        kill {PID}

        A user can also just clear out the cron script with,

        crontab -e 
        Supercomputer: 

        HOWTO: Use Docker and Singularity Containers at OSC

        It is now possible to run Docker and Singularity containers on the Ruby, Owens and Pitzer clusters at OSC. Single-node jobs are currently supported, including GPU jobs; MPI jobs are planned for the future.

        From the Docker website:  "A container image is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings."

        This document will describe how to run Docker and Singularity containers on the Ruby, Owens and Pitzer. You can use containers from Docker Hub, Sylabs CloudSingularity Hub, or any other source. As examples we will use hello-world from Singularity Hub and ubuntu from Docker Hub.

        Getting help

        The most up-to-date help on Singularity comes from the command itself.

        singularity help
        

        User guides and examples can be found at Sylabs.io.

        Setting up your environment for Singularity usage

        No setup is required. You can use Singulairty directly on all clusters.

        Accessing a container

        A Singularity container is a single file with a .sif  extension.

        * IMPORTANT NOTE: Prior to version 3.0,  Singularity uses .simg or .img as a single file extesnion when you pull out a container from a hub.

        You can simply download ("pull") a container from a hub. Popular hubs are Docker Hub and Singularity Hub. You can go there and search if they have a container that meets your needs. Docker Hub has more containers and may be more up to date but supports a much wider community than just HPC. Singularity Hub is for HPC, but the number of available containers are fewer. Additionally there are domain and vendor repositories such as biocontainers and NVIDIA HPC containers that may have relevant containers.

        Pull a container from hubs

        Docker Hub

        Pull from the 7.2.0 branch of the gcc repository on Docker Hub. The 7.2.0 is called a tag.

        singularity pull docker://gcc:7.2.0
        

        Filename:  gcc_7.2.0.sif

        Pull an Ubuntu container from Docker Hub.

        singularity pull docker://ubuntu:18.04
        

        Filename:  ubuntu_18.04.sif

        Singularity Hub

        Pull the singularityhub/hello-world ontainer from the Singularity hub. Since no tag is specified it pulls from the master branch of the repository.

        singularity pull shub://singularityhub/hello-world
        

        Filename:  hello-world_latest.sif

        Downloading containers from the hubs is not the only way to get one. You can, for example get a copy from your colleague's computer or directory. If you would like to create your own container you can start from the user guide below. If you have any questions, please contact OSC Help.

        Running a container

        There are four ways to run a container under Singularity.

        You can do this either in a batch job or on a login node. (Don’t run on a login node if the container will be performing heavy computation, of course.)

        We note that the operating system on Owens is Red Hat:

        [owens-login01]$ cat /etc/os-release
        NAME="Red Hat Enterprise Linux Server"
        VERSION="7.5 (Maipo)"
        ID="rhel"
        [..more..]

        In the examples below we will often check the operating system to show that we are really inside a container.

        Run container like a native command

        If you simply run the container image it will execute the container’s runscript.

        Example:  Run singularityhub/hello-world

        Note that this container returns you to your native OS after you run it.

        [owens-login01]$ ./hello-world_latest.sif
        Tacotacotaco
        

        Use the “run” sub-command

        The Singularity “run” sub-command does the same thing as running a container directly as described above. That is, it executes the container’s runscript.

        Example:  Run a container from a local file

        [owens-login01]$ singularity run hello-world_latest.sif
        Tacotacotaco
        

        Example:  Run a container from a hub without explicitly downloading it

        [owens-login01]$ singularity run shub://singularityhub/hello-world
        INFO: Downloading shub image
        Progress |===================================| 100.0%
        Tacotacotaco
        

        Use the “exec” sub-command

        The Singularity “exec” sub-command lets you execute an arbitrary command within your container instead of just the runscript.

        Example:  Find out what operating system the singularityhub/hello-world container uses

        [owens-login01]$ singularity exec hello-world_latest.sif cat /etc/os-release
        NAME="Ubuntu"
        VERSION="14.04.5 LTS, Trusty Tahr"
        ID=ubuntu
        [..more..] 
        

        Use the “shell” sub-command

        The Singularity “shell” sub-command invokes an interactive shell within a container.

        Example:  Run an Ubuntu shell. Note the “Singularity” prompt within the shell.

        [owens-login01 ~]$ singularity shell ubuntu_18.04.sif
        Singularity ubuntu_18.04.sif:~> cat /etc/os-release
        NAME="Ubuntu"
        VERSION="18.04 LTS (Bionic Beaver)"
        ID=ubuntu
        [.. more ..] 
        Singularity ubuntu_18.04.sif:~> exit
        exit
        

        File system access

        When you use a container you run within the container’s environment.  The directories available to you by default from the host environment are

        • your home directory
        • working directory (directory you were in when you ran the container)
        • /fs/project
        • /fs/scratch
        • /tmp

        You can review our Available File Systems page for more details about our file system access policy.

        If you run the container within a job you will have the usual access to the $PFSDIR environment variable with adding node attribute "pfsdir" in the job request (nodes=XX:ppn=XX:pfsdir). You can access most of our file systems from a container without any special treatment.

        GPU usage within a container

        If you have a GPU-enabled container you can easily run it on Owens or Pitzer just by adding the --nv flag to the singularity exec or run command.  The example below comes from the "exec" command section of Singularity User Guide.  It runs a TensorFlow example using a GPU on Owens.  (Output has been omitted from the example for brevity.)

        [owens-login01]$ sinteractive -n 28 -g 1
        ...
        [o0756]$ git clone https://github.com/tensorflow/models.git
        [o0756]$ singularity exec --nv docker://tensorflow/tensorflow:latest-gpu \
        python ./models/tutorials/image/mnist/convolutional.py
        

        In some cases it may be necessary to bind the CUDA_HOME path and add $CUDA_HOME/lib64 to the shared library search path:

        [owens-login01]$ sinteractive -n 28 -g 1
        ...
        [o0756]$ module load cuda
        [o0756]$ export SINGULARITY_BINDPATH=$CUDA_HOME
        [o0756]$ export SINGULARITYENV_LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CUDA_HOME/lib64
        [o0756]$ singularity exec --nv my_container mycmd
        

        Build a container

        If you want to create or modify a container, you need root-like privilege on any OSC system. Alternatively you can create a container on a hub or a local computer then pull/upload it to OSC system.  

        References

         

        Supercomputer: 

        HOWTO: Use Globus (Overview)

         

        GlobusOnline-265x258.jpg

        Globus is a cloud-based service designed to let users move, share, and discover research data via a single interface, regardless of its location or number of files or size.

        Globus was developed and is maintained at the University of Chicago and is used extensively at supercomputer centers and major research facilities.

        Globus is available as a free service that any user can access. More on how Globus works can be found on the Globus "How It Works" page.

        OSC has upgraded from Globus v5.3 to v5.4 endpoint, which includes features provided by Globus v4 endpoint. OSC will deprecate Globus v4 endpoint on Tuesday, Jan 26, 2021. Please start to use Globus v5.4 endpoint for transferring and sharing data.
        Please contact OSC Help if you need to transfer protected data, such as PHI and PII data into OSC HPC systems

        Further Reading

        HOWTO: Transfer files using Globus

        Globus is a reliable, high-performance file transfer platform allowing users to transfer large amounts of data seamlessly between systems. It aims to make transfers a "click-and-forget" process by setting up configuration details in the background and automating fault recovery.  

        Globus can be used for both file transfers between OSC and:

        • A computing institution with Globus installed (check with your site provider for availability) or
        • A personal computer (known as a personal endpoint)

        Users transferring between OSC and another computing institution with​ Globus installed do not need to install Globus Connect Personal, and can skip to Usage.

        If you are looking to transfer smaller sized files you can utilize OnDemand's file transfer capabilities, or use an SFTP client to connect to  sftp.osc.edu . Our general recommendation is that for small files - measured in MB to several hundred MB - to use OnDemand or SFTP. You can continue to use SFTP and get reasonable performance up to file sizes of several GB. For transfers of several GB or larger, you should consider using Globus.

        Install Globus Connect Personal

        To use Globus to transfer from a personal computer, you will need to install the Globus Connect Personal client on your computer following the steps below. Those transferring between OSC and another computing institution can skip to Usage.

        1. Login to Globus. If your institution does not have an organizational login, you may choose to either Sign in with Google or Sign in with ORCiD iD. b.png                                              
        2. Download the Globus Connect Personal Client 
          • In the main Globus web page, click "ENDPOINTS" on the sidebar.
          • Click "Create a personal endpoint" on the top-right of the pageendpoint_new.png
          • Choose a unique name for your endpoint and generate the setup key. Download the Globus Connect client for your operating systemcreate.png
        3. Install Globus Connect Personal Client
          • Windows
            1. Run the Installer
            2. Copy-Paste the setup key to complete the installation
          • Mac​​
            1. Mount your drives
            2. Copy the Globus Client to your Application Folder
            3. Start The Globus Client, and enter the provided setup key
          • Linux
            1. Un-tar the .tgz file with the command tar -zxvf
            2. Run globusconnect , found within the unzipped directory
            3. Copy-Paste the setup key when prompted
        4. (Optional) Changing directories accessible to Globus

        By default, Globus will only add certain default folders to the list of files and directories accessible by Globus. To change/add/remove files and directories from this list:

        Windows

        1. Start Globus Connect Personal
        2. Go to "Options"
        • Add directories/files using the  "+" button
        • Remove directories/files using the "-" button
        • Revert to the default accessible directories/files using the "Reset to Defaults" button
        • Any changes you make are not made permanent until you press the "Save" button

        ​​Mac

        1. Start Globus Connect Personal
        2. Go to Preferences -> Access
        • Add directories/files using the  "+" button
        • Remove directories/files using the "-" button
        • Revert to the default accessible directories/files using the "Reset to Defaults" button
        • Any changes you make are not made permanent until you press the "Save" button

        ​Linux

        1. Configure the   ~/.globusonline/lta/config-paths  file. This file is a plain text file, with each line corresponding to the configuration of a particular directory path you wish to make accessible. Each line consists of 3 comma-separated fields as below
          <path1>,<sharing flag>,<R/W flag>
          <path2>,<sharing flag>,<R/W flag>
          <path3>,<sharing flag>,<R/W flag>
          ...
          

          Path: an absolute directory/path to be permitted. A leading tilde "~" can be used to represent the home directory of the user that runs globusconnectpersonal.

          Sharing Flag: it controls sharing, with a value of "1" allowing sharing for the path and a value of "0" disallowing sharing for the path.

          R/W Flag: it determines whether the path will be accessible read-only or read/write, with a "1" permitting read/write access and a "0" marking the path as read-only. 

        2. After making a change to the ~/.globusonline/lta/config-paths file, you must stop and restart Globus Connect Personal before the changes will take effect as below
        $ ./globusconnectpersonal -stop
        $ ./globusconnectpersonal -start &
        

        Usage

        We provide instructions on how to transfer data using Globus Connect Server v5 on this page. If you would like to share your data with your collaborators, please see this page

        If you are doing a large transfer you should transfer to/from the parallel file system for best performance. 
        1. Login to Globus. If your institution does not have an organizational login, you may choose to either Sign in with Google or Sign in with ORCiD iD.b.png
        2. If you do not log in with your OSC HPC credentials, map your OSC credentials to your Globus account following this page. Select 'Ohio Supercomputer Center (OSC)' as the identity provider. You will be redirected to the page below. Provide your OSC HPC credentials: OSC_login.png

        3. Navigate to the "File Manager" on the sidebar. Select the two-panel view by switching the icon beside "Panels" on the top-right of the page if the view is one panel switch.png
        4. Enter your endpoint in one of the collections
          • If transferring to a computer with Globus Connect Personal installed, make sure to have Globus Connect Personal running. The name will be the unique name chosen during installation. It is also listed under "Administered by You" under the "ENDPOINTS" on the sidebar
        5. Refer to the table below to search the other endpoint (please copy and paste all keywords for accurate search)
          •   Endpoint Keywords to search
            OSC's home directory OSC $HOME OSC,home,netapp
            OSC's project directory OSC /fs/project OSC,project,gpfs
            OSC's scratch directory OSC /fs/scratch OSC,scratch,gpfs
            OSC's ess storage OSC /fs/ess OSC,ess,gpfs
            AWS S3 storage OSC S3 OSC,s3
          • If needed, enter your username and password for the authentication
          • You should see the files in the collection. You can also change the directory by entering the appropriate path in the Path boxtransfer.png
        6. You can now transfer files and directories both ways by selecting them and pressing the arrow indicating which way you'd like to transferstart.png
        7. Once you click 'Start', an activity report will appear as shown below, and you can click on it to see the detailsstatus.png
        8. Click "View details', you will get the activity report about the transfer. You will also receive an email once the transfer is completed (successful or failed)
        Once a transfer has begun, you do not need to keep the Globus webpage up, but you will need to make sure the Globus Connect Personal Client is running on your computer until it has completed.  If the transfer is interrupted for any reason, Globus will attempt to re-initiate the transfer automatically.

        HOWTO: Use AWS S3 in Globus

        Globus Connect Server v5 allows OSC clients to connect to Amazon S3. Please follow the steps below:

        Create a New Bucket

        • Log into Amazon AWS
        • Navigate to Services -> S3
        • Click 'Create Bucket'
        • Give the bucket a name then click 'Next'
        • Click 'Next' and then 'Next' again then 'Create bucket' to create a new bucket

        Create a New IAM

        • Log into Amazon AWS
        • Navigate to Services -> IAM

        Add Policy

        • Select Policies and then click 'Create Policy'
        • Choose the JSON tab and paste the JSON policy as below (It will be necessary to replace the value for 'Resource' to match bucket name from previous steps):
        You may also refer to this page provided by Globus for the JSON policy.
        {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Sid": "AllBuckets",
                    "Effect": "Allow",
                    "Action": [
                        "s3:ListAllMyBuckets",
                        "s3:GetBucketLocation"
                    ],
                    "Resource": "*"
                },
                {
                    "Sid": "Bucket",
                    "Effect": "Allow",
                    "Action": [
                        "s3:ListBucket"
                    ],
                    "Resource": "arn:aws:s3:::osc-globus-test"
                },
                {
                    "Sid": "Objects",
                    "Effect": "Allow",
                    "Action": [
                        "s3:DeleteObject",
                        "s3:GetObject",
                        "s3:PutObject"
                    ],
                    "Resource": "arn:aws:s3:::osc-globus-test/*"
               }
            ]
        }
        • Choose 'Review Policy'
        • Fill in the name such as 'AllowGlobusS3_osc-globus-test', click 'Create Policy' to create the new policy

        Add Group

        • Choose 'Groups' in left menu bar then click 'Create New Group'. Provide the group name such as “s3_osc-globus-test” then click 'Next Step' button
        • In search bar for Filter type 'Globus' and check box next to 'AllowGlobusS3_osc-globus-test' then click 'Next Step'
        • Click 'Create Group' after reviewing new group information. 

        Add User

        • Choose 'Users' in left menu bar and then click 'Add user'. Provide the username, check 'Programmatic access' and then click 'Next: Permissions'
        • Choose 'Add user to group', filter for the group created previous and check box next to 's3_osc-globus-test', click 'Next: Tags' and then 'Next: Review'
        • Click 'Create user'
        • Make a copy of 'Access key ID' and click 'Show' for 'Secret access key' and save both for future reference
        • Click 'Close'

        Create a New Collection

        • Login to Globus. If your institution does not have an organizational login, you may choose to either Sign in with Google or Sign in with ORCiD iD
        • Navigate to the 'ENDPOINTS' on the sidebar and search 'OSC, s3'. Click 'OSC S3' to go to this gateway
        • Click the 'Collections' menu. You will see all of the collections added by you before. To add a new collection, click 'Add a Collection'. Enter your OSC username and password for the authentication
        • Provide 'Access key' and "Access secret' from previous steps and associate Globus identity with your OSC identity
        • Provide bucket name for 'Base Directory'
        • Provide the name of the collection in 'Collection Display Name' field
        • Click 'Create Collection' to finish the creation
        • Click the logo of 'Globus' and navigate back to the 'ENDPOINTS' on the sidebar. Click the 'Administered by You' and then you can locate the new collection you just created. 

        Further Reading

         

        HOWTO: Share data using Globus

        Globus Connect Server v5 allows OSC users to share data with their collaborators who do not have OSC HPC account (the collaborator needs to sign up for a free Globus account though). The advantage of data sharing via Globus is that you do not have to move your data in order to share it. You can select directory paths to be securely shared with your collaborator, and grant them read-only or read-write access.

        Note: You can only share directories, not individual files. 

        Sharing data with collaborators 

        • Login to Globus. If your institution does not have an organizational login, you may choose to either Sign in with Google or Sign in with ORCiD iDb.png
        • Map your OSC credentials to your Globus account following this page if you haven't done this before. Select 'Ohio Supercomputer Center (OSC)' as the identity provider from the list. You will be redirected to the page below. Provide your OSC HPC credentials:OSC_login.png
        • Navigate to the "ENDPOINTS" on the sidebar. Refer to the table below to search the endpoint (please copy and paste all keywords for accurate search)
          Endpoint Keywords to search
        OSC's home directory OSC $HOME OSC,home,netapp
        OSC's project directory OSC /fs/project OSC,project,gpfs
        OSC's scratch directory OSC /fs/scratch OSC,scratch,gpfs
        OSC's ess storage OSC /fs/ess OSC,ess,gpfs
        AWS S3 storage OSC S3 OSC,s3
        • Click the endpoint. You will be redirected to the overview page of this endpoint. Click the 'Collections' menu. You will see all of the collections added under this endpoint before. To add a new collection, click 'Add a Collection'. 
        • Create a guest collection. (Note: the default path will be $HOME for home directory, /fs/project for project filesystem, /fs/scratch for scratch filesystem, and /fs/ess for ess storage. You can change to a more specific directory by providing the path in ‘Directory’). Click 'Create Collection' to finish the creation.arcum
        • Navigate to the "File Manager" on the sidebar. Select the view to be one-panel by switching the icon beside "Panels" on the top-right of the pageswitch.png
        • Enter the collection you want to share by clicking 'select a collection': choose_sharing.png
        • Click 'Your Collections' to choose the right collection choose_s.png
        • Click 'Permissions' button in the right side panepermission.png
        • To share with your collaborators, click 'Add Permissions - Share With'. You can specify the directory you want to share, choose to share with users by selecting their email address or Globus username, or share with a group or all Globus users or everyone. You can also select 'Send Email' to send email to your collaborator. If you want your collaborator to also be able to transfer data into the directory, you can also provide write access by checking the 'write' button. Once the options have been selected, click 'Add Permission'. You should be redirected to the information page on this collection, and see the people you have shared it with.
          • You can repeat this process for any number of collaborators.sharewith.pngsharewith_2.png
        • To terminate the sharing, you can click the 'X' next to the invitee on the screen.Remove.png
        • To see all collections you have shared, go to 'Endpoints' in the left bar, then 'Shareable by You'.allshare.png
        If your collaborator has the write access, he/she can transfer data and delete files within that directory and any subdirectories. So be careful about providing write access.
        It is highly recommended that you terminate the sharing or delete the collection when your collaborator has completed downloading the data. 

        Information for your collaborators 

        1. Your collaborator needs to sign up for a free Globus account for data sharing
        2. Your collaborator needs to install the Globus Connect Personal client on his/her personal computer if he/she needs to download the data. See this page for more information. 
        3. If you select 'Send Email' to send an email to your collaborator, he/she will get an email including the URL to access the share. Your collaborator can click on the link and log in with the credentials
        4. Your collaborator can also access to the share by logging into Globus. Navigate to the "ENDPOINTS" on the sidebar and click on 'Shared with You'. 

        Further Reading

         

          HOWTO: Add InCommon Authentication to Globus

          (OPTIONAL) Adding InCommon Authentication 

          Adding InCommon authentication to your Globus account allows you to login to Globus Online using your university credentials.  Using this process you can store your Globus username password for safe keeping, and instead use your university username and password to login.  If your already logged in to your university authentication system, logging in to Globus can be as simple as two clicks away.

          To use this feature, your university needs to be a InCommon participant.  Some Ohio universities active in InCommon include: Ohio State University, Case Western University, Columbus State Community College, Miami University, Ohio Northern University, Ohio University, University of Findlay, University of Dayton, and many more.  

          For a complete list, visit https://incommon.org/participants/ .

          To add InCommon Authentication:
          1. Login to Globus Online
          2. Go to "Manage Identities" under your username
          3. Click "Add External Identity"
            • Choose a name for the Identity Settings. 
            • Choose InCommon / CILogon from the drop down menu
          4. On the next page, choose your University / Identity Provider
          • Click "Remember this selection"
          • Click "Log on"
          • You may be prompted to login to your university authentication system if you are not already

          When you go to login next, click "alternative login" and then "InCommon / CILogon".  Select your university on the next page, and login using your university credentials.  Globus will remember this preference, and automatically prompt you to login using your university authentication next time.

          HOWTO: Deploy your own endpoint on a server

          OSC clients who are affiliated with Ohio State can deploy their own endpoint on a server using OSU subscriptions. Please follow the steps below:

          1. Send a request to OSC Help the following information:
            • Name of organization that will be running the endpoint, ie: OSU Arts and Sciences
              • NOTE: if the name already exists, they will have to coordinate with the existing Admin for that project
            • OSU affiliated email address associated with the Globus account, ie: name.#@osu.edu
          2. OSC will create a new project at https://developers.globus.org, make the user provided in #1 the administrator, and inform the user to set up the endpoint credentials
          3. The user goes to https://developers.globus.org/ and chooses “Register a new Globus Connect Server v5”. Under the project, the user chooses Add dropdown and chooses Add new Globus Connect Server. Provide a display name for the endpoint, ie: datamover02.hpc.osc.edu. Select “Generate New Client Secret” and save that value and Client ID and use those values when configuring the Globus Connect Server install on their local system
          4. The user finishes configuring Globus Connect Server and runs the necessary commands to register the new endpoint with Globus. Once the new endpoint is registered, please email OSC Help the endpoint name so we can mark the endpoint as managed under the OSU subscription

           

          Supercomputer: 

          HOWTO: Use VNC in a batch job

          SSHing directly to a compute node at OSC - even if that node has been assigned to you in a current batch job - and starting VNC is an "unsafe" thing to do. When your batch job ends (and the node is assigned to other users), stray processes will be left behind and negatively impact other users. However, it is possible to use VNC on compute nodes safely.

          You can use OnDemand, which is a much easier way to access desktops. If your work is not a very large, very intensive computation (for example, you do not expect to saturate all of the cores on a machine for a significant portion of the time you have the application you require open - e.g., you are using the GUI to set up a problem for a longer non-interactive compute job), you can choose one VDI under "Virtual Desktop Interface" from "Desktops" menu. Otherwise, please use "Interactive HPC" from Desktops" menu.

          The examples below are for Pitzer. If you use other systems, please see this page for supported versions of TurboVNC on our systems.

          Starting your VNC server

          Step one is to create your VNC server inside a batch job.

          Option 1: Interactive

          The preferred method is to start an interactive job, requesting an gpu node, and then once your job starts, you can start the VNC server.

          salloc --nodes=1 --ntasks-per-node=40 --gpus-per-node=1 --gres=vis --constraint=40core srun --pty /bin/bash
          

          This command requests an entire GPU node, and tells the batch system you wish to use the GPUs for visualization. This will ensure that the X11 server can access the GPU for acceleration. In this example, I have not specified a duration, which will then default to 1 hour.

          module load virtualgl
          module load turbovnc
          

          Then start your VNC server. (The first time you run this command, it may ask you for a password - this is to secure your VNC session from unauthorized connections. Set it to whatever password you desire. We recommend a strong password.)

          vncserver
          
          To set the vnc password again use the vncpasswd command.

          The output of this command is important: it tells you where to point your client to access your desktop. Specifically, we need both the host name (before the :), and the screen (after the :).

          New 'X' desktop is p0302.ten.osc.edu:1
          

          Connecting to your VNC server

          Because the compute nodes of our clusters are not directly accessible, you must log in to one of the login nodes and allow your VNC client to "tunnel" through SSH to the compute node. The specific method of doing so may vary depending on your client software.

          The port assigned to the vncserver will be needed. It is usually 5900 + <display_number>. e.g.

          New 'X' desktop is p0302.ten.osc.edu:1

          would use port 5901.

          Linux/MacOS

          Option 1: Manually create an SSH tunnel 

          I will be providing the basic command line syntax, which works on Linux and MacOS. You would issue this in a new terminal window on your local machine, creating a new connection to Pitzer.

          ssh -L <port>:<node_hostname>.ten.osc.edu:<port> <username>@pitzer.osc.edu
          

          The above command establishes a proper ssh connection for the vnc client to use for tunneling to the node.

          Open your VNC client, and connect to localhost:<screen_number>, which will tunnel to the correct node on Pitzer.

          Option 2: Use your VNC software to tunnel 

          This example uses Chicken of the VNC, a MacOS VNC client. It is a vncserver started on host n0302 with port 5901 and display 1.

          The default window that comes up for Chicken requires the host to connect to, the screen (or port) number, and optionally allows you to specify a host to tunnel through via SSH. This screenshot shows a proper configuration for the output of vncserver shown above. Substitute your host, screen, and username as appropriate.

          When you click [Connect], you will be prompted for your HPC password (to establish the tunnel, provided you did not input it into the "password" box on this dialog), and then (if you set one), for your VNC password. If your passwords are correct, the desktop will display in your client.

          Windows

          This example shows how to create a SSH tunnel through your ssh client.  We will be using Putty in this example, but these steps are applicable to most SSH clients.

          First, make sure you have x11 forwarding enabled in your SSH client.

          Next, open up the port forwarding/tunnels settings and enter the hostname and port you got earlier in the destination field.  You will need to add 5900 to the port number when specifiying it here.  Some clients may have separate boxes for the desination hostname and port.  

          For source port, pick a number between 11-99 and add 5900 to it.  This number between 11-99 will be the port you connect to in your VNC client.

          Make sure to add the forwaded port, and save the changes you've made before exiting the configutations window.

          PuTTY Tunnel Configuration Settings

          Now start a SSH session to the respective cluster your vncserver is running on.  The port forwarding will automatically happen in the background.  Closing this SSH session will close the forwarded port; leave the session open as long as you want to use VNC.

          Now start a VNC client.  TurboVNC has been tested with our systems and is recommended.  Enter localhost:[port], replacing [port] with the port between 11-99 you chose earlier.

          New TurboVNC Connection

          If you've set up a VNC password you will be prompted for it now.  A desktop display should pop up now if everything is configured correctly.

          How to Kill a VNC session?

          Occasionally you may make a mistake and start a VNC server on a login node or somewhere else you did not want to.  In this case it is important to know how to properly kill your VNC server so no processes are left behind.

          The command syntax to kill a VNC session is:

          vncserver -kill :[screen]
          

          In the example above, screen would be 1.

          You need to make sure you are on the same node you spawned the VNC server on when running this command.

          Supercomputer: 
          Service: 
          Fields of Science: 

          HOWTO: Use a Conda Environment With Jupyter

          Before Conda environments may be used by Jupyter Notebooks they must be installed so that Jupyter knows about them. Older versions of Conda automatically installed a Jupyter kernel for themselves, that installation process now must be performed manually by the user.

          To perform the installation the user should load their prefered version of Python, activate their Conda environment and run the following command:

          NOTE: The ipykernel package should be installed in the conda environment first!
          conda install -n $MYENV ipykernel
          python -m ipykernel install --user --name "$MYENV" --display-name "Python ($MYENV)"

          Where the variable MYENV is replaced with the name of the user's Conda environment. A fuller example:

          module load python
          MYENV='useful-project-name'
          source activate "$MYENV"
          python -m ipykernel install --user --name "$MYENV" --display-name "Python ($MYENV)"

          If the envirnoment is rebuilt or renamed, you may want to erase the installation of a custom jupyter kernel.

          This command will erase entire directories and the files within them. Be careful!
          rm -rf ~/.local/share/jupyter/kernels/$(MYENV)
          Supercomputer: 
          Service: 

          HOWTO: Use an Externally Hosted License

          Many software packages require a license.  These licenses are usually made available via a license server, which allows software to check out necessary licenses.  In this document external refers to a license server that is not hosted inside OSC.

          If you have such a software license server set up using a license manager, such as FlexNet, this guide will instruct you on the necessary steps to connect to and use the licenses at OSC.

          Users who wish to host their software licenses inside OSC should consult OSC Help.

           

          You are responsible for ensuring you are following your software license terms.  Please ensure your terms allow you to use the license at OSC before beginning this process!

          Introduction

          Broadly speaking, there are two different ways in which the external license server's network may be configured.  These differ by whether the license server is directly externally reachable or if it sits behind a private internal network with a port forwarding firewall.  

          If your license server sits behind a private internal network with a port forwarding firewall you will need to take additional steps to allow the connection from our systems to the license server to be properly routed. 

          License Server is Directly Externally Reachable

          Diagram showing network setup of license server that is directly externally reachable

           

          License Server is Behind Port Forwarding Firewall

          Diagram showing network setup of license server that is behind a port forwarding firewall

           

          Unsure?

          If you are unsure about which category your situation falls under contact your local IT administrator.

          Configure Remote Firewall

          In order for connections from OSC to reach the license server, the license server's firewall will need to be configured.  All outbound network traffic from all of OSC's compute nodes are routed through a network address translation host (NAT), or two backup servers.

          The license server should be configured to allow connections from the following IP addresses to the SERVER:PORT where the license server is running:

          • nat.osc.edu (192.157.5.13)
          • 192.148.248.35
          • 192.148.248.186

          A typical FlexNet-based license server uses two ports: one is server port and the other is daemon port, and the firewall should be configured for the both ports. A typical license file looks, for example,

          SERVER licXXX.osc.edu 0050XXXXX5C 28000
          VENDOR {license name} port=28001
          

          In this example, "28000" is the server port, and "28001" is the daemon port. The daemon port is not mandatory if you use it on a local network, however it becomes necessary if you want to use it outside of your local network. So, please make sure you declared the daemon port in the license file and configured the firewall for the port.

          Confirm Configuration

          The firewall settings should be verified by attempting to connect to the license server from the compute environment using telenet.

          Get on to a compute node by requesting a short, small, interactive job and test the connection using telenet:

          telnet <License Server IP Address> <Port#>
          

          (Recommended) Restrict Access to IPs/Usernames

          It is also recommended to restrict accessibility using the remote license server's access control mechanisms, such as limiting access to particular usernames in the options.dat file used with FlexNet-based license servers.

          For FlexNet tools, you can add the following line to your options.dat file, one for each user.

          INCLUDEALL USER <OSC username>
          

          If you have a large number of users to give access to you may want to define a group using GROUP within the options.dat file and give access to that whole group using INCLUDEALL GROUP <group name> .

          Users who use other license managers should consult the license manager's documentation.

          Modify Job Environment to Point at License Server

          The software must now be told to contact the license server for it's licenses.  The exact method of doing so can vary between each software package, but most use an environment variable that specifies the license server IP address and port number to use.

          For example LS DYNA uses the environment variable LSTC_LICENSE and LSTC_LICENSE_SERVER to know where to look for the license.  The following lines would be added to a job script to tell LS-DYNA to use licenses from port 2345 on server 1.2.3.4, if you use bash:

          export LSTC_LICENSE=network
          export LSTC_LICENSE_SERVER=2345@1.2.3.4
          

          or, if you use csh:

          setenv LSTC_LICENSE network
          setenv LSTC_LICENSE_SERVER 2345@1.2.3.4
          

          License Server is Behind Port Forwarding Firewall

          If the license server is behind a port forwarding firewall, and has a different IP address from the IP address of the firewall, additional steps must be taken to allow connections to be properly routed within the license server's internal network.  

          1. Use the license server's fully qualified domain name in SERVER line in the license file instead of the IP address.
          2. Contact your IT team to have the firewall IP address mapped to the fully qualified domain name.

           

          Software Specific Details

          The following outlines details particular to a specific software package.  

          ANSYS

          Uses the following environment variables:

          
          ANSYSLI_SERVERS=<port>@<IP>
          ANSYSLMD_LICENSE_FILE=<port>@<IP>
          

          If your license server is behind a port forwarding firewall and you cannot use a fully qualified domain name in the license file, you can add ANSYSLI_EXTERNAL_IP={external IP address} to ansyslmd.ini on the license server.

           

           

          HOWTO: Use ulimit command to set soft limits

          This document shows you how to set soft limits using ulimit command.

          The ulimit command sets or reports user process resource limits. The default limits are defined and applied when a new user is added to the system. Limits are categorized as either soft or hard. With the ulimit command, you can change your soft limits for the current shell environment, up to the maximum set by the hard limits. You must have root user authority to change resource hard limits.

          Syntax

          ulimit [-H] [-S] [-a] [-c] [-d] [-f] [-m] [-n] [-s] [-t] [Limit]
          flags description

          -a

          Lists all of the current resource limits

          -c

          Specifies the size of core dumps, in number of 512-byte blocks

          -d

          Specifies the size of the data area, in number of K bytes

          -f

          Sets the file size limit in blocks when the Limit parameter is used, or reports the file size limit if no parameter is specified. The -f flag is the default

          -H

          Specifies that the hard limit for the given resource is set. If you have root user authority, you can increase the hard limit. Anyone can decrease it

          -m

          Specifies the size of physical memory, in number of K bytes

          -n

          Specifies the limit on the number of file descriptors a process may have

          -s

          Specifies the stack size, in number of K bytes

          -S

          Specifies that the soft limit for the given resource is set. A soft limit can be increased up to the value of the hard limit. If neither the -H nor -S flags are specified, the limit applies to both

          -t

          Specifies the number of seconds to be used by each process

          The limit for a specified resource is set when the Limit parameter is specified. The value of the Limit parameter can be a number in the unit specified with each resource, or the value "unlimited". For example, to set the file size limit to 51,200 bytes, use:

          ulimit -f 100

          To set the size of core dumps to unlimited, use:

          ulimit –c unlimited

          How to change ulimit for a MPI program

          ulimit command affects the current shell environment. When a MPI program is started, it does not spawn in the current shell. You have to use mpiexec to start a wrapper script that sets the limit if you want to set the limit for each process. Below is how you set the limit for each shell (We use  ulimit –c unlimited to allow unlimited core dumps, as an example): 

          1. Prepare your batch job script named "myjob" as below (Here, we request a job with 5-hour 2-node):
          #!/bin/bash
          #SBATCH --nodes=2
          #SBATCH --time=5:00:00
          #SBATCH ...
          
          
          ...
          mpiexec ./test1
          ...
          1. Prepare the wrapper script named "test1" as below:
          #!/bin/bash
          ulimit –c unlimited
          .....(your own program)
          1. sbatch myjob
          Supercomputer: 
          Service: 

          Citation

          The Ohio Supercomputer Center provides High Performance Computing resources and expertise to academic researchers across the State of Ohio. Any paper citing this document has utilized OSC to conduct research on our production services. OSC is a member of the Ohio Technology Consortium, a division of the Ohio Department of Higher Education.

          Hardware

          OSC services can be cited by visiting the documentation for the service in question and finding the "Citation" page (located in the menu to the side).

          HPC systems currently in production use can be found here: https://www.osc.edu/supercomputing/hpc

          Decommissioned HPC systems can be found here: https://www.osc.edu/supercomputing/hpc/decommissioned.

          Logos

          Please refer to our branding webpage

          Citing OSC

          We prefer that you cite OSC when using our services, using the following information, taking into account the appropriate citation style guidelines. For your convenience, we have included the citation information in BibTeX and EndNote formats.

          Ohio Supercomputer Center. 1987. Ohio Supercomputer Center. Columbus OH: Ohio Supercomputer Center. http://osc.edu/ark:/19495/f5s1ph73.

          BibTeX:

          @misc{OhioSupercomputerCenter1987,
          ark = {ark:/19495/f5s1ph73},
          url = {http://osc.edu/ark:/19495/f5s1ph73},
          year  = {1987},
          author = {Ohio Supercomputer Center},
          title = {Ohio Supercomputer Center}
          }
          

          EndNote:

          %0 Generic
          %T Ohio Supercomputer Center
          %A Ohio Supercomputer Center
          %R ark:/19495/f5s1ph73
          %U http://osc.edu/ark:/19495/f5s1ph73
          %D 1987

          Here is an .ris file to better suit your needs. Please change the import option to .ris

          Documentation Attachment: 

          OSC Custom Commands

          There are some commands that OSC has created custom versions of to be more useful to OSC users.

           

          OSCfinger

          Introduction

          OSCfinger is a command developed at OSC for use on OSC's systems and is similar to the standard finger command. It allows various account information to be viewed.

          Availability

          owens PITZER
          X

          X

           

          Usage

          OSCfinger takes the following options and parameters.

          $ OSCfinger -h
          usage: OSCfinger.py [-h] [-e] [-g] USER
          
          positional arguments:
            USER
          
          optional arguments:
            -h, --help   show this help message and exit
            -e           Extend search to include gecos/full name (user) or
                         category/institution (group)
            -g, --group  Query group instead of users
          
          Query user:
              OSCfinger foobar
          
          Query by first or last name:
              OSCfinger -e Foo
              OSCfinger -e Bar
          
          Query group:
              OSCfinger -g PZS0001
          
          Query group by category or insitituion:
              OSCfinger -e -g OSC    

           

          View information by username

          The OSCfinger command can be used to view account information given a username.

          $ OSCfinger jsmith
          Login: xxx                                   Name: John Smith
          Directory: xxx                               Shell: /bin/bash
          E-mail: xxx
          Primary Group: PPP1234
          Groups: 
          

          Project Information by Project ID

          The OSCfinger command can also reveal details about a project using the -g flag.

          $ OSCfinger -g PPP1234
          Group: PPP1234                                    GID: 1234
          Status: ACTIVE                                    Type: Academic
          Principal Investigator: xxx                       Admins: NA
          Members: xxx
          Category: NA
          Institution: OHIO SUPERCOMPUTER CENTER
          Description: xxx
          ---
          

          Search for a user via first and/or last name

          If the username is not known, a lookup can be initiated using the -e flag.

          This example is shown using the lookup for a first and last name.

          $ OSCfinger -e "John Smith"
          Login: jsmith                                     Name: John Smith
          Directory: xxx                                    Shell: /bin/bash
          E-mail: NA
          Primary Group: PPP1234
          Groups: xxx
          Password Changed: Jul 04 1776 15:47 (calculated)  Password Expires: Aug 21 1778 12:05 AM
          Login Disabled: FALSE                             Password Expired: FALSE
          ---
          

           

          One can also lookup users with only the last name:

          $ OSCfinger -e smith
          Login: jsmith                                      Name: John Smith
          Directory: xxx                                    Shell: /bin/bash
          E-mail: NA
          Primary Group: PPP1234
          Groups:
          ---
          Login: asmith                                     Name: Anne Smith
          Directory: xxx                                    Shell: /bin/bash
          E-mail: xxx
          Primary Group: xxx
          Groups: 
          ---
          

           

          Only the first name can also be used, but many accounts are likely to be returned.

          $ OSCfinger -e John
          Login: jsmith                                     Name: John Smith
          Directory: xxx                                    Shell: /bin/bash
          E-mail: xxx
          Primary Group: PPP1234
          Groups:
          ---
          Login: xxx                                        Name: John XXX
          Directory: xxx                                    Shell: /bin/bash
          E-mail: xxx
          Primary Group: xxx
          Groups:
          ---
          Login: xxx                                        Name: John XXX
          Directory: xxx                                    Shell: /bin/ksh
          E-mail: xxx
          Primary Group: xxx
          Groups:
          ---
          ...(more accounts below)...
          

          Slurm usage

          While in a slurm environment, the OSCfinger command shows some additional information:

          $ OSCfinger jsmith 
          Login: xxx Name: John Smith 
          Directory: xxx Shell: /bin/bash 
          E-mail: xxx 
          Primary Group: PPP1234 
          Groups:
          SLURM Enabled: TRUE
          SLURM Clusters: pitzer
          SLURM Accounts: PPP1234, PPP4321
          SLURM Default Account: PPPP1234
          

          It's important to note that the default account in slurm will be used if an account is not specified at job submission.

          Supercomputer: 
          Service: 

          OSCgetent

          Introduction

          OSCgetent is a command developed at OSC for use on OSC's systems and is similar to the standard getent command. It lets one view group information.

          Availability

          owens PITZER
          X

          X

           

          Usage

          OSCgetent takes the following options and parameters.

          $ OSCgetent -h
          usage: OSCgetent.py [-h] {group} [name [name ...]]
          
          positional arguments:
            {group}
            name
          
          optional arguments:
            -h, --help  show this help message and exit
          
          Query group:
              OSCgetent.py group PZS0708
          
          Query multiple groups:
              OSCgetent.py group PZS0708 PZS0709
              

          View group information

          The OSCgetent command can be used to view group(s) members:

          $ OSCgetent group PZS0712
          PZS0712:*:5513:amarcum,amarcumtest,amarcumtest2,guilfoos,hhamblin,kcahill,xwang 
          

          View information on multiple groups

          $ OSCgetent group PZS0712 PZS0708
          PZS0708:*:5509:djohnson,ewahl,kearley,kyriacou,linli,soottikkal,tdockendorf,troy
          PZS0712:*:5513:amarcum,amarcumtest,amarcumtest2,guilfoos,hhamblin,kcahill,xwang

           

          Supercomputer: 
          Service: 

          OSCprojects

          Introduction

          OSCprojects is a command developed at OSC for use on OSC's systems and is used to view your logged in accounts project information.

          Availability

          owens PITZER
          X

          X

           

          Usage

          OSCprojects does not take any arguments or options:

          $ OSCprojects
          OSC projects for user amarcumtest2:
          
          Project         Status          Members
          -------         ------          -------
          PZS0712         ACTIVE          amarcumtest2,amarcumtest,guilfoos,amarcum,xwang
          PZS0726         ACTIVE          amarcumtest2,xwangtest,amarcum
          

          This command returns the current users projects, whether those projects are active/restricted and the current members of the projects.

          Supercomputer: 
          Service: 

          OSCusage

          Introduction

          OSCusage is command developed at OSC for use on OSC's systems.  It allows for a user to see information on their project's usage, including different users and their jobs.

          Availability

          owens PITZER
          X X

           

          Usage

          OSCusage takes the following options and parameters.

          $ OSCusage --help
          usage: OSCusage.py [-h] [-u USER]
                             [-s {opt,pitzer,glenn,bale,oak,oakley,owens,ruby}] [-A]
                             [-P PROJECT] [-q] [-r] [-n] [-v]
                             [start_date] [end_date]
          
          positional arguments:
            start_date            start date (default: 2020-09-13)
            end_date              end date (default: 2020-09-14)
          
          optional arguments:
            -h, --help            show this help message and exit
            -u USER, --user USER  username to run as. Be sure to include -P or -A.
                                  (default: amarcum)
            -s {opt,pitzer,glenn,bale,oak,oakley,owens,ruby}, --system {opt,pitzer,glenn,bale,oak,oakley,owens,ruby}
            -A                    Show all
            -P PROJECT, --project PROJECT
                                  project to query (default: PZS0712)
            -q                    show user data
            -r                    show raw
            -n                    show job ID
            -v                    do not summarize
          
          Usage Examples:
          
              Specify start time:
                  OSCusage 2018-01-24
          
              Specify start and end time:
                  OSCusage 2018-01-24 2018-01-25
          

           

          Today's Usage

          Running OSCusage with no options or parameters specified will provide the usage information in Dollars for the current day.

          $ OSCusage
          ----------------  ------------------------------------
                            Usage Statistics for project PZS0712
          Time              2020-09-13 to 2020-09-14
          PI                Brian Guilfoos <guilfoos@osc.edu>
          Remaining Budget  -1.15
          ----------------  ------------------------------------
          
          User          Jobs    core-hours    Status
          ------------  ------  ------------  ----------
          amarcum       0       0.0           ACTIVE
          amarcumtest   0       0.0           ACTIVE
          amarcumtest2  0       0.0           RESTRICTED
          guilfoos      0       0.0           ACTIVE
          hhamblin      0       0.0           ACTIVE
          kcahill       0       0.0           ACTIVE
          wouma         0       0.0           ACTIVE
          xwang         0       0.0           ACTIVE
          --            --      --
          TOTAL         0       0.0
          

          Usage in Timeframe

          If you specify a timeframe you can get utilization information specifically for jobs that completed within that period.

          $ OSCusage 2020-01-01 2020-07-01
          ----------------  ------------------------------------
                            Usage Statistics for project PZS0712
          Time              2020-01-01 to 2020-07-01
          PI                Brian Guilfoos <guilfoos@osc.edu>
          Remaining Budget  -1.15
          ----------------  ------------------------------------
          
          User          Jobs    core-hours    Status
          ------------  ------  ------------  ----------
          amarcum       86      260.3887      ACTIVE
          amarcumtest   0       0.0           ACTIVE
          amarcumtest2  0       0.0           RESTRICTED
          guilfoos      9       29.187        ACTIVE
          hhamblin      1       1.01          ACTIVE
          kcahill       7       40.5812       ACTIVE
          wouma         63      841.2503      ACTIVE
          xwang         253     8148.2638     ACTIVE
          --            --      --
          TOTAL         419     9320.681
          

          Show only a single user's usage

          Specify -q to show only the current user's usage. This stacks with -u to specify which user you want to see.

          $ OSCusage -u xwang -q 2020-01-01 2020-07-01
          ----  -------------------------------
                Usage Statistics for user xwang
          Time  2020-01-01 to 2020-07-01
          ----  -------------------------------
          
          User    Jobs    core-hours    Status
          ------  ------  ------------  --------
          xwang   253     8148.2638     -
          --      --      --
          TOTAL   253     8148.2638

          Show a particular project

          By default, the tool shows your default (first) project. You can use -P to specify which charge code to report on.

          $ OSCusage -P PZS0200
          ----------------  ------------------------------------
                            Usage Statistics for project PZS0200
          Time              2020-09-13 to 2020-09-14
          PI                David Hudak <dhudak@osc.edu>
          Remaining Budget  0
          ----------------  ------------------------------------
          
          User        Jobs    core-hours    Status
          ----------  ------  ------------  ----------
          adraghi     0       0.0           ARCHIVED
          airani      0       0.0           ARCHIVED
          alingg      0       0.0           ARCHIVED

           

          You can show all of your charge codes/projects at once, by using -A .

          Select a particular cluster

          By default, all charges are shown in the output. However, you can filter to show a particular system with -s .

          $ OSCusage -s ruby
          ----------------  ------------------------------------
                            Usage Statistics for project PZS0712
          Time              2020-09-13 to 2020-09-14
          PI                Brian Guilfoos <guilfoos@osc.edu>
          Remaining Budget  -1.15
          ----------------  ------------------------------------
          
          User          Jobs    core-hours    Status
          ------------  ------  ------------  ----------
          amarcum       0       0.0           ACTIVE
          amarcumtest   0       0.0           ACTIVE
          amarcumtest2  0       0.0           RESTRICTED
          guilfoos      0       0.0           ACTIVE
          hhamblin      0       0.0           ACTIVE
          kcahill       0       0.0           ACTIVE
          wouma         0       0.0           ACTIVE
          xwang         0       0.0           ACTIVE
          --            --      --
          TOTAL         0       0.0

          Changing the units reported

          The report shows usage in core-hours. You can elect to get usage in raw seconds using -r

          $ OSCusage 2020-01-01 2020-07-01 -r
          ----------------  ------------------------------------
                            Usage Statistics for project PZS0712
          Time              2020-01-01 to 2020-07-01
          PI                Brian Guilfoos <guilfoos@osc.edu>
          Remaining Budget  -1.15
          ----------------  ------------------------------------
          
          User          Jobs    raw_used    Status
          ------------  ------  ----------  ----------
          amarcum       86      937397.0    ACTIVE
          amarcumtest   0       0.0         ACTIVE
          amarcumtest2  0       0.0         RESTRICTED
          guilfoos      9       105073.0    ACTIVE
          hhamblin      1       3636.0      ACTIVE
          kcahill       7       146092.0    ACTIVE
          wouma         63      3028500.0   ACTIVE
          xwang         253     29333749.0  ACTIVE
          --            --      --
          TOTAL         419     33554447.0
          
          
          Detailed Charges Breakdown

          Specify -v to get detailed information jobs.

          You can add the -n option to the -v option to add the job ID to the report output. OSCHelp will need the job ID to answer any questions about a particular job record.

          Please contact OSC Help with questions. 

          Supercomputer: 
          Service: 

          checkgpu Command

          Introduction

          checkgpu is a command developed at OSC for use on OSC’s systems to report information regarding the usage of OSC’s GPU nodes. It reports various information on the usage and availability of the GPU nodes.

          Availability

          owens PITZER
          X

          X

           

          Usage

          checkgpu takes the following options and parameters (viewable by passing  -h/--help)

          $ checkgpu -h
          usage: checkgpu.py [-h] [-j] [-r] [-q] [-n] [-u] [-a] [-v]
          
          optional arguments:
            -h, --help     show this help message and exit
            -j, --jobs     Check status of gpu jobs
            -r, --run      Check status of running gpu jobs (use with --jobs)
            -q, --queued   Check status of queued gpu jobs (use with --jobs)
            -n, --node     Check status of gpu nodes
            -u, --used     Check status of used gpu nodes (use with --node)
            -a, --avail    Check status of available gpu nodes (use with --node)
            -v, --verbose  Display full job report
          

          View information on GPU jobs

          By using the flag -j/--jobs, checkgpu can report summaries for the total number of jobs using GPU nodes

          $ checkgpu -j
          *** SUMMARY ***
          ===================================================
          Summary of Running GPU Jobs:
          
          152 GPU jobs running using 152 GPU nodes. Among them:
           * 146 GPU jobs running using 146 GPU nodes from non-condo and non-debugging jobs
           * 0 GPU jobs running using 0 GPU nodes from debugging jobs
           * 6 GPU Jobs running using 6 GPU nodes from PCONXXXX
          ===================================================
          Summary of Queued GPU Jobs:
          
          485 GPU Jobs queued requesting 492 GPU nodes. Among them:
           * 382 GPU Jobs queued requesting 389 GPU nodes from non-condo and non-debugging jobs
           * 0 GPU Jobs queued requesting 0 debug GPU nodes
           * 103 GPU Jobs queued requesting 103 GPU nodes from PCONXXXX
          ===================================================
          

          View information on GPU jobs (run-only)

          By using the flag -r/--run in tandem with -j, checkgpu can report summaries for the total number of running jobs using GPU nodes

          $ checkgpu -j --run
          ===================================================
          Summary of Running GPU Jobs:
          152 GPU jobs running using 152 GPU nodes. Among them:
           * 146 GPU jobs running using 146 GPU nodes from non-condo and non-debugging jobs
           * 0 GPU jobs running using 0 GPU nodes from debugging jobs
           * 6 GPU Jobs running using 6 GPU nodes from PCONXXXX

          View information on GPU jobs (queue-only)

          By using the flag-q/--queued in tandem with -j, checkgpu can report summaries for the total number of queued jobs using GPU nodes

          $ checkgpu -j --queued
          ===================================================
          Summary of Queued GPU jobs:
          484 GPU Jobs queued requesting 491 GPU nodes. Among them:
           * 382 GPU Jobs queued requesting 389 GPU nodes from non-condo and non-debugging jobs
           * 0 GPU Jobs queued requesting 0 debug GPU nodes
           * 102 GPU Jobs queued requesting 102 GPU nodes from PCONXXXX

          View information on GPU nodes

          By using the flag -n/--node, checkgpu can report summaries for the usage of OSC’s GPU nodes

          $ checkgpu --node
          ***USED NODES***
          ================================================================
          ================================================================
          Summary of the current GPU nodes:
          153 total jobs on gpu nodes
           * 153 GPU jobs using 153 GPU nodes. Among them:
            ** 147 GPU nodes used by non-condo and non-debug jobs
            ** 0 GPU nodes used by debug jobs
            ** 6 GPU nodes used by condo group PCONXXXX
           * 0 non-GPU jobs using 0 GPU nodes. Among them:
            ** 0 GPU nodes used by non-condo and non-debug non-gpu jobs
            ** 0 GPU nodes used by debug non-gpu jobs
            ** 0 GPU nodes used by non-gpu condo jobs
          
          ***AVAILABLE NODES***
          ================================================================
          ================================================================
          Summary of the available GPU nodes:
          144 available gpu nodes. Among them: 
           * 4 fully available gpu nodes.
           * 140 partly available gpu nodes.
           * 0 open gpus on partly available nodes.
          

          View information on GPU nodes (used-only)

          By using the flag -u/--used in tandem with -n, checkgpu can report summaries for the usage of used GPU nodes

          $ checkgpu -n -u
          ================================================================
          Summary of the current GPU nodes:
          152 total jobs on gpu nodes
           * 152 GPU jobs using 152 GPU nodes. Among them:
            ** 147 GPU nodes used by non-condo and non-debug jobs
            ** 0 GPU nodes used by debug jobs
            ** 5 GPU nodes used by condo group PCONXXXX
           * 0 non-GPU jobs using 0 GPU nodes. Among them:
            ** 0 GPU nodes used by non-condo and non-debug non-gpu jobs
            ** 0 GPU nodes used by debug non-gpu jobs
            ** 0 GPU nodes used by non-gpu condo jobs

          View information on GPU nodes (available-only)

          By using the flag -a/--avail in tandem with -n, checkgpu can report summaries for the availability of GPU nodes

          $ chekgpu -n -a
          ================================================================
          Summary of the available GPU nodes:
          144 available gpu nodes. Among them: 
           * 4 fully available gpu nodes.
           * 140 partly available gpu nodes.
           * 0 open gpus on partly available nodes.
          

          View full job summary

          By using the flag -v/--verbose in tandem with any of the above command options, checkgpu will report full summaries on the usage of GPU nodes

          $ checkgpu --jobs --run -v
          ============================================
          Job Details:
          List of non-condo and non-debug GPU jobs
          XXXXXXX.owens-batch.ten.osc.edu
          o0678-gpu/0
          XXXXXXX.owens-batch.ten.osc.edu
          o0673-gpu/0
          XXXXXXX.owens-batch.ten.osc.edu
          o0695-gpu/0
          XXXXXXX.owens-batch.ten.osc.edu
          o0805-gpu/0
          XXXXXXX.owens-batch.ten.osc.edu
          o0763-gpu/0
          ...
          
          List of debug GPU jobs
          
          List of GPU jobs from PCON0005
          XXXXXXX.owens-batch.ten.osc.edu
          o0732-gpu/0
          XXXXXXX.owens-batch.ten.osc.edu
          o0779-gpu/0
          XXXXXXX.owens-batch.ten.osc.edu
          o0745-gpu/0
          XXXXXXX.owens-batch.ten.osc.edu
          o0712-gpu/0
          XXXXXXX.owens-batch.ten.osc.edu
          o0663-gpu/0
          XXXXXXX.owens-batch.ten.osc.edu
          o0750-gpu/0
          
          ===================================================
          Summary of Running GPU Jobs:
          153 GPU jobs running using 153 GPU nodes. Among them:
           * 147 GPU jobs running using 147 GPU nodes from non-condo and non-debugging jobs
           * 0 GPU jobs running using 0 GPU nodes from debugging jobs
           * 6 GPU Jobs running using 6 GPU nodes from PCON0005
          Supercomputer: 
          Service: 

          OSC User Code of Ethics

          The Ohio Supercomputer Center (OSC) exists to provide state-of-the-art computing services to universities and colleges; to provide supercomputer services to Ohio scientists and engineers; to stimulate unique uses of supercomputers in Ohio; to attract students, faculty, resources and industry; to catalyze inter-institutional supercomputer research and development projects; to serve as the model for other state-sponsored technology initiatives.

          OSC serves a large number and variety of users including students, faculty, staff members, and commercial clients throughout the state of Ohio.  Ethical and legal standards, in particular, that apply to the use of computing facilities are not unique to the computing field. Rather, they derive directly from standards of common sense and common decency that apply to the use of any public resource.  Indeed, OSC depends upon the spirit of mutual respect and cooperative attitudes.

          This statement on conditions of use is published in that spirit. The purpose of this statement is to promote the responsible, ethical, and secure use of OSC resources for the protection of all users.

          Authorized Use

          As a condition of use of OSC facilities, the user agrees:

          1. To respect the privacy of other users; for example, users shall not intentionally seek information on, obtain copies of, or modify files, tapes, or passwords belonging to other users unless explicitly authorized to do so by those users.
          2. To respect the legal protection provided by copyrights and licenses to programs and data; for example, users shall not make copies of a licensed computer program to avoid paying additional license fees.
          3. To respect the intended usage for which access to computing resources was granted; for example, users shall use accounts authorized for their use by the principal investigator responsible for these accounts only for the purposes specified by the principal investigator and shall not use any other user's account.
          4. To respect the integrity of computing systems; for example, users shall not intentionally develop or use programs that harass other users or infiltrate a computer or computing systems or damage or alter the software components of a computing system.
          5. To respect the financial structure of computing systems; for example, users shall not intentionally develop or use any unauthorized mechanisms to alter or avoid charges levied by OSC for computing services.
          6. To not enable other institutions or users to avoid licensing restrictions or fees by simply allowing them to use their account.
          7. To abide by software specific licensing terms and restrictions, as outlined in the specific software page or agreement.

          In addition, users are expected to report to OSC information that they may obtain concerning instances in which the above conditions have been or are being violated.

          Violations of the following conditions are certainly unethical and are possibly a criminal offense: unauthorized use of another user's account; tampering with other users' files, tapes, or passwords, harassment of other users; unauthorized alteration of computer charges; and unauthorized copying or distribution of copyrighted or licensed software or data. Therefore, when OSC becomes aware of possible violations of these conditions, it will initiate an investigation. At the same time, in order to prevent further possible unauthorized activity, OSC may suspend the authorization of computing services to the individual or account in question. In accordance with established practices, confirmation of the unauthorized use of the facilities by an individual may result in disciplinary review, expulsion from his/her university, termination of employment, and/or legal action.

          Users of computing resources should be aware that although OSC provides and preserves the security of files, account numbers, and passwords, security can be breached through actions or causes beyond reasonable control. Users are urged, therefore, to safeguard their data, to take full advantage of file security mechanisms built into the computing systems, and to change account passwords frequently.

          Appropriate Use

          Computing resources shall be used in a manner consistent with the instructional and/or research objectives of the community, in general, and consistent with the objectives of the specified project for which such use was authorized. All uses inconsistent with these objectives are considered to be inappropriate use and may jeopardize further authorization.

          Beyond the allocation of computing resources, OSC normally cannot and does not judge the value or appropriateness of any user's computing. However, the use of computing resources for playing games for purely recreational purposes, the production of output that is unrelated to the objectives of the account, and, in general, the use of computers simply to use computing resources are examples of questionable use of these resources.

          When possible inappropriate use of computing resources is encountered, OSC shall notify the principal investigator responsible. The principal investigator is expected either to take action or to indicate that such use should be considered appropriate.

          Should possible inappropriate use continue after notification of the principal investigator, or should unresolvable differences of opinion persist, these shall be brought to the attention of OSC staff for recommendations on further action.  Upon the recommendation of OSC staff, the Director may impose limitations on continued use of computing resources.

          Responsible Use

          Users are expected to use computing resources in a responsible and efficient manner consistent with the goals of the account for which the resources were approved. OSC will provide guidance to users in their efforts to achieve efficient and productive use of these resources. Novice users may not be aware of efficient and effective techniques; such users may not know how to optimize program execution; nor may such optimization necessarily lead to improved cost benefits for these users. Those who use large amounts of computing resources in production runs should attempt to optimize their programs to avoid the case where large inefficient programs deny resources to other users.

          Programming, especially in an interactive environment, involves people, computers, and systems. Efficient use of certain resources, such as computers, may lead to inefficient use of other resources, such as people. Indeed, the benefits attributed to good personal or interactive computing systems are that they speed total program development and thus lower attendant development costs even though they may require more total computer resources. Even with this understanding, however, users are expected to refrain from engaging in deliberately wasteful practices, for example, performing endless unnecessary computations.

          OSC Responsibilities

          OSC has a responsibility to provide service in the most efficient manner that best meets the needs of the total user community. At certain times the process of carrying out these responsibilities may require special actions or intervention by the staff. At all other times, OSC staff members have no special rights above and beyond those of other users. OSC shall make every effort to ensure that persons in positions of trust do not misuse computing resources or take advantage of their positions to access information not required in the performance of their duties.

          OSC prefers not to act as a disciplinary agency or to engage in policing activities. However, in cases of unauthorized, inappropriate, or irresponsible behavior the Center does reserve the right to take action, commencing with an investigation of the possible abuse. In this connection, OSC, with all due regard for the rights of privacy and other rights of users', shall have the authority to examine files, passwords, accounting information, printouts, tapes, or other material that may aid the investigation. Examination of users files must be authorized by the Director of OSC or his designee. Users, when requested, are expected to cooperate in such investigations. Failure to do so may be grounds for cancellation of access privileges.


          OSC User Code of Ethics © 2018

          Supercomputing FAQ

          General Questions

          Account Questions

          Disk Storage Questions

          Email Questions

          Linux Questions

          SSH Questions

          Batch Processing Questions

          Compiling System Questions

          Parallel Processing Questions

          Libraries/Software Questions

          Performance Analysis Questions

          Other Common Problems

          General Questions

          Who can get an account?

          Anyone can have an account with OSC, but you need access to a project to utilize our resources. If an eligible principal investigator has a current project, he/she can add the user through client protal my.osc.edu. Authorized users do not have to be located in Ohio. See https://www.osc.edu/supercomputing/support/account.

          Where should a new OSC user begin?

          Once you are able to connect to our HPC systems, you should start familiarizing yourself with the software and services available from the OSC, including:

          Do I have to pay for supercomputer use?

          It depends on the type of client and your rate of consumption. Please click here for more information.

          How many supercomputers does OSC have?  Which one should I use?

          OSC currently has two HPC clusters: Pitzer Cluster, a 29,664 core Dell cluster with Intel Xeon proccessors, and Owens Cluster, a 23,500+ core Dell cluster with Intel Xeon processors. New users have access to Pitzer and Owens clusters. To learn more, click here.

          How do I cite OSC in my publications?

          Any publication of any material, whether copyrighted or not, based on or developed with OSC services, should cite the use of OSC, and the use of the specific services (where applicable). For more information about citing OSC, please visit www.osc.edu/citation.

          How do I submit my publications and funding information to OSC?

          You can add these to your profile in MyOSC: https://www.osc.edu/supercomputing/portals/client_portal/manage_profile_information 

          You can then associate them with OSC project(s). 

          Can I rceive a letter of support from OSC when I apply for outside funding?

          OSC has a standard letter of support that you can include (electronically or in hard copy) with a proposal for outside funding. This letter does not replace the budget process. To receive the letter of support, please send your request to oschelp@osc.edu. You should provide the following information: name and address of the person/organization to whom the letter should be addressed; name(s) of the principal investigator(s) and the institution(s); title of the proposal; number of years of proposed project; budget requested per year. Please allow at least two working days to process your request.

          Hardware information about the systems is available at http://www.osc.edu/supercomputing/hardware

          How do I register for a workshop?

          For a complete schedule of current training offerings, please visit the OSC Training Schedule.  To register or for more information, please email oschelp@osc.edu.  

          Where can I find documentation?

          For documentation specific to software applications, see Software.  For other available hardware, see Supercomputers.

          My question isn't answered here. Whom can I ask for help?

          Contact the OSC Help Desk. Support is available 24x7x365, but more complicated questions will need to wait for regular business hours (Monday - Friday, 9am - 5pm) to be addressed. More information on the OSC supercomputing help desk can be found on our Support Services page.

          Something seems to be wrong with the OSC systems.  Should I contact the help desk?

          Information will be coming soon for guidelines on reporting possible system problems.

          Where can I find logos for my presentations, posters, etc.?

          Please see our citation webpage.

          Account Questions

          What are projects and accounts?

          An eligible principal investigator heads a project. Under a project, authorized users have accounts with credentials that permit users to gain access to the HPC systems. A principal investigator can have more than one project. 

          How do I get/renew an account?

          For information concerning accounts (i.e., how to apply, who can apply, etc.), see Accounts.

          I'm a faculty member.  How do I get accounts for my students?

          If an eligible principal investigator is new to OSC, he/she can create a new project. If an eligible principal investigator has a current project, he/she can add the user through client protal my.osc.edu. Authorized users do not have to be located in Ohio.

          ADD FEE STRUCTURE FAQ

          I'm continuing the research of a student who graduated.  Can I use his/her account?

          Please have your PI send an email to oschelp@osc.edu for further discussions. 

          I'm working closely with another student.  Can we share an account?

          No. Each person using the OSC systems must have his/her own account. Sharing files is possible, even with separate accounts. 

          How do I change my password?

          You can change your password through the MyOSC portal. Log in at http://my.osc.edu, and click the "change password" button at the bottom left corner of the HPC User Profile box. Please note that your password has certain requirements; these are specified on the "change password" portal. Please wait 3-5 minutes before attempting to log in with the new password. For security purposes, please note that our password change policy requires a password change every 180 days.

          If your password has expired, you can update by following the "Forgot your password?" link at my.osc.edu login page.

          I want to use csh instead of bash.  How do I change the default shell?

          You can change your default shell through the MyOSC portal. Log in at my.osc.edu, and use the "Unix Shell" drop-down menu in the HPC User Profile box to change your shell. You will need to log off the HPC system and log back on before the change goes into effect. Please note, that it will take about a few minutes for the changes to be applied.

          How do I find my project budget balance?

          To see usage and balance information from any system, refer to the OSCusage page.

          NOTE: Accounting is updated once a day, so the account balance is for the previous day.

          How do I get more resources?

          To request additional use of our resources, the principal investigator will need to change the budget for their project. Please see the creating budgets and projects page.

          How much will my project be charged for supercomputer usage?

          If the project is associated with an Ohio academic institution, see the academic fee structure page for pricing.

          If the project is NOT associated with an Ohio academic institution, contact OSC Sales for information on pricing.

          See Job and storage charging for how OSC calculates charges.

          Disk Storage Questions

          What is my disk quota?

          Each user has a quota of 500 gigabytes (GB) of storage and 1,000,000 files. You may also have access to a project directory with a separate quota.  See Available File Systems for more information.

          How can I determine the total disk space used by my account?

          Your quota and disk usage are displayed every time you log in. You have limits on both the amount of space you use and the number of files you have. There are separate quotas for your home directory and any project directories you have access to.

          Note: The quota information displayed at login is updated twice a day, so the information may not reflect the curent usage.

          You may display your home directory quota information with quota -s .

          How do I get more disk space?

          Your home directory quota cannot be increased. You should consider deleting, transferring, and/or compressing your files to reduce your usage. 

          A PI may request project space to be shared by all users on a project. Estimate the amount of disk space that you will need and the duration that you will need it. Send requests to oschelp@osc.edu.

          How can I find my largest directories?

          To reveal the directories in your account that are taking up the most disk space you can use the du , sort and tail commands. For example, to display the ten largest directories, change to your home directory and then run the command:

          du . | sort -n | tail -n 10
          

          Why do I receive "no space left" error when writing data to my home directory?

          If you receive the error "No space left on device" when you try to write data to your home directory, it indicates the disk is full. First, check your home directory quota. Each user has 500 GB quota of storage and the quota information is shown when you login to our systems. If your disk quota is full, consider reducing your disk space usage. If your disk quota isn't full (usage less than 500GB), it is very likely that your disk is filled up with 'snapshot' files, which are invisible to users and used to track fine-grained changes to your files for recovering lost/deleted files. In this case, please contact OSC Help for further assitance. To avoid this situation in future, consider running jobs that do a lot of disk I/O in the temporary filesystem ($TMPDIR or $PFSDIR) and copy the final output back at the end of the run. See Available File Systems for more information.

          How can I use tar and gzip to aggregate and compress my files?

          The commands tar and gzip can be used together to produce compressed file archives representing entire directory structures. These allow convenient packaging of entire directory contents. For example, to package a directory structure rooted at src/ use

          tar -czvf src.tar.gz src/
          

          This archive can then be unpackaged using

          tar -xzvf src.tar.gz
          

          where the resulting directory/file structure is identical to what it was initially.

          The programs zip , bzip2 and compress can also be used to create compressed file archives. See the man pages on these programs for more details.

          Tar is taking too long.  Is there a way to compress quicker?

          If using tar with the options zcvf is taking too long you can instead use pigz  in conjunction with tar.   pigz does gzip compression while taking advantage of multiple cores.

          tar cvf - paths-to-archive | pigz > archive.tgz
          

          pigz defaults to using eight cores, but you can have it use more or less with the -p argument.

          tar cvf - paths-to-archive | pigz -n 4 > archive.tgz
          
          Due to the parallel nature of pigz, if you are using it on a login node you should limit it to using 2 cores.  If you would like to use more cores you need to submit either an interactive or batch job to the queue and do the compression from within the job.

          Note: pigz does not significantly improve decompression time.  

          Email Questions

          How do I change the email address OSC uses to contact me?

          Please update your email on my.osc.edu, or send your new contact information to oschelp@osc.edu

          I got an automated email from OSC.  Where can I get more information about it?

          See the Knowledge Base.

          Linux Questions

          What is Linux?

          Linux is an open-source operating system that is similar to UNIX. It is widely used in High Performance Computing.

          How can I get started using Linux?

          See the Unix Basics tutorial for more information.  There are also many tutorials available on the web.

          SSH Questions

          What is SSH?

          Secure Shell (SSH) is a program to log into another computer over a network, to execute commands in a remote machine, and to move files from one machine to another. It provides strong authentication and secure communications over insecure channels. SSH provides secure X connections and secure forwarding of arbitrary TCP connections.

          How does SSH work?

          SSH works by the exchange and verification of information, using public and private keys, to identify hosts and users. The  ssh-keygen command creates a directory ~/.ssh and files that contain your authentication information. The public key is stored in ~/.ssh/id_rsa.pub and the private key is stored in  ~/.ssh/id_rsa. Share only your public key. Never share your private key. To further protect your private key you should enter a passphrase to encrypt the key when it is stored in the file system. This will prevent people from using it even if they gain access to your files. 

          One other important file is ~/.ssh/authorized_keys. Append your public keys to the authorized_keys file and keep the same copy of it on each system where you will make ssh connections. 

          In addition, on Owens the default SSH client config enables hashing of a user’s known_hosts file.  So if SSH is used on Owens the remote system’s SSH key is added to ~/.ssh/known_hosts in a hashed format which can’t be unhashed.  If the remote server’s SSH key changes, special steps must be taken to remove the SSH key entry:

          ssh-keygen -R <hostname>
          

          Can I connect without using an SSH client?

          The OSC OnDemand portal allows you to connect to our systems using your web browser, without having to install any software. You get a login shell and also the ability to transfer files.  

          How can I upload or download files?

          Most file transfers are done using sftp (SSH File Transfer Protocol) or scp (Secure CoPy). These utilities are usually provided on Linux/UNIX and Mac platforms. Windows users should read the next section, "Where can I find SSH and SFTP clients".

          Where can I find SSH and SFTP clients?

          There are many SSH and SFTP clients available, both commercial and free.  See Getting Connected for some suggestions.  

          How do I run a graphical application in an SSH session?

          Graphics are handled using the X11 protocol. You’ll need to run an X display server on your local system and also set your SSH client to forward (or "tunnel") X11 connections. On most UNIX and Linux systems, the X server will probably be running already. On a Mac or Windows system, there are several choices available, both commercial and free. See our guide to Getting Connected for some suggestions.

          Why do I get "connection refused" when trying to connect to a cluster?

          OSC temporarily blacklists some IP addresses when multiple failed logins occur. If you are connecting from behind a NAT gateway, as is commonly used for public or campus wireless networks, and get a "connection refused" message it is likely that someone recently tried to connect multiple times and failed when connected to the same network you are on. Please contact OSC Help with your public IP address and the cluster you attempted to connect to and we will remove your IP from the blacklist. You can learn your public IP by searching for "what is my IP address" in Google.

          Batch Processing Questions

          What is a batch request?

          On all OSC systems, batch processing is managed by the Simple Linux Utility for Resource Management system (SLURM). SLURM batch requests (jobs) are shell scripts that contain the same set of commands that you enter interactively. These requests may also include options for the batch system that provide timing, memory, and processor information. For more information, see our guide to Batch Processing at OSC.  

          How do I submit, check the status, and/or delete a batch job?

          SLURM uses sbatch  to submit, squeue to check the status, and scancel  to delete a batch request. For more information, see our Batch-Related Command Summary

          Can I be notified by email when my batch job starts or ends?

          Yes.  See the --mail-type  option in our SLURM docoumentation. If you are submitting a large number of jobs, this may not be a good idea.

          Why won't my job run?

          There are numerous reasons why a job might not run even though there appear to be processors and/or memory available. These include:

          • Your account may be at or near the job count or processor count limit for an individual user.
          • Your group/project may be at or near the job count or processor count limit for a group.
          • The scheduler may be trying to free enough processors to run a large parallel job.
          •  Your job may need to run longer than the time left until the start of a scheduled downtime.
          • You may have requested a scarce resource or node type, either inadvertently or by design.

          See our Scheduling Policies and Limits for more information. 

          How can I retrieve files from unexpectedly terminated jobs?

          A batch job that terminates before the script is completed can still copy files from $TMPDIR to the user's home directory via the trap command ( trap commands do not work in csh and tcsh shell batch scripts). In the batch script, the trap command needs to precede the command causing the TERMination. It could be placed immediately after the SLURM header lines. Here is a generic form:

          trap "cd $SLURM_SUBMIT_DIR;mkdir $SLURM_JOB_ID;cp -R $TMPDIR/* $SLURM_JOB_ID;exit" TERM
          

          If a command in a batch script is killed for excessive memory usage (see Out-of-Memory (OOM) or Excessive Memory Usage for details) then the trap command may not be executed.  However, normal shell scripting can handle this situation:  the exit status of a command that may possibly cause an OOM can be checked and appropriate action taken.  Here is a Bourne shell example:

          bla_bla_big_memory_using_command_that_may_cause_an_OOM
          if [ $? -ne 0 ]; then
          cd $SLURM_SUBMIT_DIR;mkdir $SLURM_JOB_ID;cp -R $TMPDIR/* $SLURM_JOB_ID
          exit
          fi
          

          Finally, if a node your job is running on crashes then the trap command may not be executed. It may be possible to recover your files from batch-managed directories in this case. Contact OSC Help for assistance.

          How can I delete all of my jobs on a cluster?

          To delete all your jobs on one of the clusters, including those currently running, queued, and in hold, login to the cluster and run the command:

          squeue -u <username> | xargs scancel
          

          How can I determine the number of cores in use by me or my group?

          To be provided later

          How to request GPU nodes for visualization?

          By default, we don't start an X server on gpu nodes because it impacts computational performance. Add vis  in your GPU request such that the batch system uses the GPUs for visualization. For example, on Owens, it should be

          --nodes=1 --ntasks-per-node=28 --gpus-per-node=1 --gres=vis
          

          Compiling System Questions

          What languages are available?

          Fortran, C, and C++ are available on all OSC systems. The commands used to invoke the compilers and/or loaders vary from system to system. For more information, see our Compilation Guide.

          What compiler (vendor) do you recommend?

          We have Intel, PGI, and gnu compilers available on all systems. Each compiler vendor supports some options that the other doesn’t, so the choice depends on your individual needs. For more information, see our Compilation Guide.

          Will software built for one system run on another system?

          Most serial code built on one system will run on another system, although it may run more efficiently if it is built and run on the same system. Parallel (MPI) code typically must be built on the system where it will run.

          Parallel Processing Questions

          What is parallel processing?

          Parallel processing is the simultaneous use of more than one computer (or processor) to solve a problem. There are many different kinds of parallel computers. They are distinguished by the kind of interconnection between processors or nodes (groups of processors) and between processors and memory.

          What parallel processing environments are available?

          On most systems, both shared-memory and distributed-memory parallel programming models can be used. Versions of OpenMP (for multithreading or shared-memory usage) and MPI (for message-passing or distributed-memory usage) are available.  A summary of parallel environments will be coming soon.

          What is a core?

          A core is a processor. When a single chip contains multiple processors, they are called cores.

          I'm not seeing the performance I expected.  How can I be sure my code is running in parallel?

          We are currently working on a guide for this. Please contact OSC Help for assistance.

          Libraries/Software Questions

          What software applications are available?

          See the Software section for more information.

          Do you have a newer version of (name your favorite software)?

          Check the Software section to see what versions are installed. You can also check the installed modules using the module spider  or module avail  command.

          How do I get authorized to use a particular software application?

          Please contact OSC Help for assistance.

          What math routines are available?  Do you have ATLAS and LAPACK?

          See the Software section for information on third-party math libraries (e.g., MKL, ACML, fftw, scalapack, etc). MKL and ACML are highly optimized libraries that include the BLAS and LAPACK plus some other math routines. 

          Do you have NumPy/SciPy?

          The NumPy and SciPy modules are installed with the python software. See the Python software page.

          OSC does not have a particular software package I would like to use. How can I request it?

          Please refer to the Software Forms page. You will see a link to Request for Software Form. Download the form, complete the information, and attach the form to an e-mail to oschelp@osc.edu. The Statewide Users Group will consider the request.

          You may install open source software yourself in your home directory. If you have your own license for commercial software, contact the OSC Help desk.

          I have a software package that must be installed as root.  What should I do?

          Most packages have a (poorly documented) option to install under a normal user account. Contact the OSC Help desk if you need assistance. We generally do not install user software as root.

          What are modules?

          Modules are used to manage the environment variable settings associated with software packages in a shell-independent way. On OSC's systems, you will by default have modules in your environment for PBS, MPI, compilers, and a few other pieces of software. For information on using the module system, see our guide to Batch Processing at OSC.

          Performance Analysis Questions

          What are MFLOPS/GFLOPS/TFLOPS/PFLOPS?

          MegaFLOPS/GigaFLOPS/TeraFLOPS/PetaFLOPS are millions/billions/trillions/quadrillions of FLoating-point Operations (calculations) Per Second.

          How do I find out about my code's performance?

          A number of performance analysis tools are available on OSC systems. Some are general to all systems and others are specific to a particular system. See our performance analysis guide for more info.

          How can I optimize my code?

          There are several ways to optimize code. Key areas to consider are CPU optimization, I/O optimization, memory optimization, and parallel optimization. See our optimization strategy guide for more info.

          Other Common Problems

          What does "CPU time limit exceeded" mean?

          Programs run on the login nodes are subject to strict CPU time limits. To run an application that takes more time, you need to create a batch request. Your batch request should include an appropriate estimate for the amount of time that your application will need. See our guide to Batch Processing at OSC for more information.

          My program or file transfer died for no reason after 20 minutes.  What happened?

          Programs run on the login nodes are subject to strict CPU time limits. Because file transfers use encryption, you may hit this limit when transferring a large file. To run longer programs, use the batch system. To transfer larger files, connect to sftp.osc.edu instead of to a login node.

          Why did my program die with a segmentation fault, address error, or signal 11?

          This is most commonly caused by trying to access an array beyond its bounds -- for example, trying to access element 15 of an array with only 10 elements. Unallocated arrays and invalid pointers are other causes. You may wish to debug your program using one of the available tools such as the TotalView Debugger.

          I created a batch script in a text editor on a Windows or Mac system, but when I submit it on an OSC system, almost every line in the script gives an error. Why is that?

          Windows and Mac have different end-of-line conventions for text files than UNIX and Linux systems do, and most UNIX shells (including the ones interpreting your batch script) don't like seeing the extra character that Windows appends to each line or the alternate character used by Mac. You can use the following commands on the Linux system to convert a text file from Windows or Mac format to UNIX format:

          dos2unix myfile.txt 
          
          mac2unix myfile.txt  
          

          I copied my output file to a Windows system, but it doesn't display correctly.  How can I fix it?

          A text file created on Linux/UNIX will usually display correctly in Wordpad but not in Notepad. You can use the following command on the Linux system to convert a text file from UNIX format to Windows format:

          unix2dos myfile.txt  
          

          What IP ranges do I need to allow in my firewall to use OSC services?

          See our knowledge base article on the topic.

          Supercomputer: 

          Supercomputing Terms

          (alphabetical listing)

          authorized users

          Authorized users include the principal investigator and secondary investigators who are part of the research team on a project. For classroom accounts, authorized users are the registered students and teaching assistants.

          authorized users, adding new ones to existing project

          To add a new authorized user to a project, the principal investigator can invite new users or add existing users through OSC client portal

          balance (budget)

          To determine your project balance (budget), please utilize MyOSC or log on to any machine and use the following command: OSCusage
          To maintain a positive balance (budget), make sure to submit new budgets using this page for guidance.

          classroom project

          A project that allows students to learn high-performance computing or to apply high-performance computing in a particular course through applications. The budget awarded is $500 and can be renewed if needed; credits cover all costs. Please see our classroom guide for more information. 

          principal investigator

          • A full-time, permanent researcher or faculty member of an Ohio college or university.
            • Responsibilities of a principal investigator
              • Central contact and administrator of the project
              • Responsible for monitoring project balance (budget) and submitting new budget applications in a timely manner
              • Ensure the ethical use of OSC's resources by the research team

          project

          A project contains one or more research activities, which may or may not be related. Each project has a number consisting of a three- or four-letter prefix and four numbers. Principal investigators may have more than one project, but they should be aware that $1,000 annual credit can only apply to one service agreement which can be applied to multiple projects.

          secondary investigators

          These are authorized users other than the principal investigator. The PI is responsible for keeping OSC updated on changes in authorized users.

          Statewide Users Group

          The Statewide Users Group comprises representatives from Ohio's colleges and universities. The members serve as an advisory body to OSC.

          support from other sources

          If your research is supported by monetary accounts from funding agencies, the Center appreciates learning of this. Such data helps the Center determine its role in Ohio's research activities.

          type of project

          The Center mainly categorizes projects as a classroom (fully subsidized) or Ohio academic ($1,000 annual grant per PI). There are other types of projects the Center may deem fit, such as commercial. 

          username

          Unique login name of a user.  Make changes to password, shell, email, project access on OSC's client portal, MyOSC (my.osc.edu). 

          Available Software

          Available Software icon

          OSC has a variety of software applications to support all aspects of scientific research. You can view the complete software list, which is being updated continually.

          Recent changes can be found by looking at the Changelog.

          OSC also offers licenses for some software packages to Ohio researchers via our statewide software program.

          Some packages are access-controlled due to license restrictions. You can find the forms necessary to request access.

          Available software by OSC System and Application

          Complete list of current software filterable by OSC system and use or field of study.

          Access Forms

          Some OSC software requires completion of a form before access can be granted.

          Statewide Software

          Statewide licensed software tools that will facilitate research.

          Software Refresh

          Information on software updates on OSC system.

          License Server Status

          Interruption details and status of the license servers.

          Browse Software

          License Server Status

          Interruption information and status of license servers are posted below. If you have any questions, please contact OSC Help .

          Current Issues

          • No active interruptions 

          Previous Issues

          • Hyperworks academic license (license6) : unstable from 10:15am EST Dec 31, 2020 Due to license updat: stable from 10:20am EST Dec 31, 2020. License updated
          • Abaqus academic license (license6) : unstable from 2:30pm EST Oct 26, 2020 Due to license update: stable from 2:40pm EST Oct 26, 2020. License updated
          • Hyperworks academic license (license6, state-wide) : unstable from 3:35pm EST Sep 9, 2020 Due to license update: stable from 4:00pm EST Sep 9, 2020. License updated
          • Abaqus academic license (license6) : unstable from 7:10pm EST May 16, 2020: stable from 7:50pm EST May 16, 2020. License restored
          • Comsol academic license (license6) : unstable from 1:25am EST May 16, 2020 Due to license maintenance: stable from 1:50am EST May 16, 2020. License updated
          • Intel compiler license (license5) : unstable from 12:00pm EST Mar 19, 2020 due to license update; stable from 12:20pm EST Mar 19, 2020. License updated
          • Ansys academic license (license5) : unstable from 9:50am EST Feb 28, 2020 Due to license server update: stable from 10:20am EST Feb 28, 2020. License server updated
          • ls-dyna academic license (license8) : unavailable from 1:30pm EST Oct 15, 2019 Due to the system maintenance. stable from 1:50pm EST Oct 16, 2019. system update is done. 
          • Ansys academic license (license5) : unstable from 3:55pm EST Oct 11, 2019 Due to license update: stable from 4:30pm EST Oct 11, 2019. License updated
          • matlab academic license (license5) : unstable from 4:15pm EST Sep 26, 2019 Due to license update: stable from 4:40pm EST Sep 26, 2019. License updated
          • Intel compiler license (license5) : unstable from 8:50am EST Sep 24, 2019 Due to license server update: stable from 9:10am EST Sep 24, 2019. License server updated
          • ls-dyna academic license (license8) : unavailable from 1:00pm EST Sep 12, 2019 Due to the system maintenance: stable from 2:20pm EST Sep 12, 2019. System updated
          • ArcGIS academic license (license5) : unstable from 11:10am EST Apr 3, 2019 Due to license update: stable from 11:30am EST Apr 3, 2019. License updated
          • Ansys academic license (license5) : unstable from 9:20am EST Mar 29, 2019 Due to license update: stable from 9:30am EST Mar 29, 2019. License updated
          • Intel compiler license (license5) : unstable from 10:40am EST Mar 13, 2019 Due to license update: stable from 11:10am EST Mar 13, 2019. License updated
          • Altair Hyperworks academic license (license6) : unstable from 10:15am EST Dec 27, 2018 Due to license renewal: stable from 11:15am EST Dec 17, 2018. License renewed
          • ls-dyna academic license (license5) : unavailable from 8:00am EST Oct 23, 2018 Due to the system maintenance: available from 12pm EST Oct 24, 2018. Maintenance finished
          • ArcGIS academic license (license5) : unavailable from 8:00am EST Oct 23, 2018 Due to the system maintenance: available from 4pm EST Oct 24, 2018. Maintenance finished
          • ls-dyna academic license (license5) : unavailable from 4:00pm EST Oct 18, 2018: available from 8:40pm EST Oct 18, 2018. The issue fixed.
          • Comsol academic license (license6) : unstable from 10:37am EST Oct 11, 2018 Due to license maintenance: stable from 11:30am EST Oct 11, 2018. license updated
          • Ansys academic license (license5) : unstable from 2:26pm EST Jul 18, 2018 Due to license maintenance: stable from 2:47pm EST Jul 18, 2018. license updated
          • license6 server has been rebooted between 9am to 9:10am on May 29 (Tuesday) 2018 due to the maintenance requirement. The affected software was academic abaqus, pgi, starccm, schrodinger, comsol, hyperworks, totalview and xfdtd. All the licenses are back online at 9:10am May 29, 2018.
          • StarCCM academic license (license6) : outage from 12:00am EST Nov 27, 2017 Due to license renewal: license recovered 11:30am EST Nov 30, 2017. License updated
          • Ansys academic license (license5) : unstable from 12:20pm EST Nov 8, 2017 Due to license maintenance: stable from 1:30pm EST Nov 8, 2017. license updated
          • intel academic license (license5) : unstable from 4:00pm EST Oct 13, 2017 Due to license update: stable from 4:30pm EST Oct 13, 2017. license updated
          • Starccm academic license (license6) : expecting outage from Sep 23, 2017 Due to license renewal: license recovered 08:00PM EST Sep 25, 2017
          • matlab academic license (license5) : unstable from 11:40am EST Jul 19, 2017 Due to license update: stable from 12:07pm EST Jul 19, 2017. license updated
          • intel academic license (license5) : unstable from 9:40am EST Jul 6, 2017 Due to license debug: stable from 11:00am EST Jul 6, 2017. Bug fixed
          • totalview and hyperworks academic license (license2) : unstable from 2:47pm EST Jun 26, 2017 Due to license migration: stable from Jun 28, 2017. Migration done
          • comsol academic license (license2) : unstable from 4:25pm EST May 16, 2017 Due to license maintenance: stable from 4:45pm EST May 16, 2017. New version installed
          • lstc academic license (license5) : unstable from 10:00am EST Jan 12, 2017 Due to license bug fixing: stable from 12:40pm EST Jan 12, 2017. Bug fixed
          • Ansys academic license (license2) : unstable from 10:15am EST Jan 11, 2017 Due to license maintenance: stable from 10:45am EST Jan 11, 2017. other maintenance expected
          • Altair Hyperworks academic license (license2) : stopped from 3:15am EST Dec 12, 2016 Due to license update: restarted by 3:25pm EST DEc 12, 2016: license updated
          • Intel compiler academic license (license5) : stopped from 11:20am EST Oct 27, 2016 Due to software update: restared by 11:40am EST Oct 27, 2016: software updated
          • abaqus academic license (license2) : stopped from 2:20pm EST Oct 26, 2016 Due to license renewal: restarted by 2:30pm EST Oct 26, 2016: license renewed
          • Matlab academic license (license5) : stopped from 4:10pm EST Sep 29, 2016 Due to license renewal: restored by 4:30pm EST Sep 29, 2016: license renewed.
          • starccm academic license (license2) : stopped from 2:00pm EST Sep 23, 2016 Due to license update: restored by 2:10pm EST Sep 23, 2016: license updated.
          • arcgis academic license (license3) : unstable from 3:45pm EST Aug 22, 2016 Due to troubleshooting: restored by 5:00pm EST Aug 22, 2016: Problem fixed, but test needed
          • Ansys academic license (license2) : unstable from 3:00am EST Aug 15, 2016 Due to troubleshooting: restored by 3:50pm EST Aug 15, 2016: Found another problem. Another attempt expected
          • Ansys academic license (license2) : stopped from 9:50am EST Aug 12, 2016 Due to software update: restarted on 11:10am EST Aug 12, 2016: Update failed. Another attempt expected
          • Comsol academic license (license2) : unstable from 3:20pm EST Aug 5, 2016 Due to troubleshooting: stable from 4:00pm EST Aug 5, 2016: Problem fixed
          • Comsol academic license (license2) : stopped from 11:10am EST Aug 5, 2016 Due to software update: restarted on 11:30am EST Aug 5, 2016. Update finished
          • arcGIS license: stopped from 11:20am EST Jul 21, 2016 Due to software update: restarted on 2:00pm EST Jul 21, 2016. Update finished
          • PGI statewide license: stopped from 11:20am EST Jul 20, 2016. Due to software update: restarted on 11:55am EST Jul 20, 2016. Update finished

           

           

           

          Scientific Database List

          This page provides a list of the scientific database available at OSC. 

          Supercomputer: 
          Service: 
          Fields of Science: 

          BLAST Database

           

          OSC periodically updates The NCBI BLAST database. 

          Versions

          BLAST database is available on the Owens and Pitzer clusters. The versions currently available at OSC are:

          Version Owens Pitzer
          2017-03 X  
          2017-10 X*  
          2018-08 X X*
          2019-09 X X
          2020-04 X X
          * Current default version

          The version indicates the date of download. You can usemodule spider blast-database​ to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          BLAST database is available to all OSC users. If you have any questions, please contact OSC Help.

          Usage

          BLAST database can be accessed with the following module:

          module load blast-database/version
          

          To list all the BLAST database available

          module spider blast-database
          
          

          BLAST database can be accessed by the environmental variable BLASTDB. For blast-database/2018-08, it is as follows

          BLASTDB=/fs/project/pub_data/blast-database/2018-08

          Further Reading

          BLAST package: https://www.osc.edu/resources/available_software/software_list/blast

          Supercomputer: 
          Service: 

          Microbial Genomes Database

          Microbial Genomes is a database of prokaryotic genome sequencing project data. 

          Availability and Restrictions

          Versions

          Microbial Genomes is available on the Owens cluster. The versions currently available at OSC are:

          Version Owens
          2017-11 X*
          * Current default version

          The version indicates the date of the installation. You can use module spider microbial-database to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          Microbial Genomes is available to all OSC users. If you have any questions, please contact OSC Help.

          Usage

          Usage on Owens

          Set-up

          To configure your enviorment for use of TrimGalore, use command module load microbial-database. This will load the default version.

          The database can be accessed by the environment variable MICROBIALDB, for example

           >> ls $MICROBIALDB
          Archaea_Genome  Bacteria_Genome  Fungi_Genome  Protozoa_Genome  Virus_Genome

          Further Reading

          Supercomputer: 
          Fields of Science: 

          Software Forms

          Several of our software packages require that users complete a form that is then kept on record here before we can grant access to the software. OSC will provide you with the proper forms and instructions; please contact OSC Help noting which software package(s) you would like access.

          The following list includes third-party software packages that require additional forms be completed:

          ABAQUS

          Amber

          ANSYS

          CASINO

          CNS

          CSD

          FLUENT

          GATK

          Gaussian

          Hyperworks

          MATLAB DCS

          MuTect

          SIESTA

          STAR-CCM+ 

          Turbomole

          Requesting additional software

          If you would like to request that the center install (and purchase, if necessary) software, please complete the Request for Software Form. The return instructions are noted on the form.

          If you have licensed software and you need assistance installing the software and restricting it's use to your group, please contact OSC Help.

          Service: 

          Software List

          Ohio Supercomputer Center (OSC) has a variety of software applications to support all aspects of scientific research. We are actively updating this documentation to ensure it matches the state of the supercomputers. This page is currently missing some content; use module spider on each system for a comprehensive list of available software.

          Supercomputer: 
          Service: 

          ABAQUS

          ABAQUS is a finite element analysis program owned and supported by SIMULIA, the Dassault Systèmes brand for Realistic Simulation.

          Availability and Restrictions

          Versions

          The available programs are ABAQUS/CAE, ABAQUS/Standard and ABAQUS/Explicit. The versions currently available at OSC are:

          Version Owens Notes
          6.13    
          6.14 X  
          2016 X Version scheme has been changed
          2017 X  
          2018 X  
          2020 X*  
          *: Default Version

          You can use  module spider abaqus to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access for Academic Users

          OSC's ABAQUS license can only be used for educational, institutional, instructional, and/or research purposes. Only users who are faculty, research staff or students at the following institutions are permitted to utilized OSC's license:

          • The Ohio State University
          • University of Toledo
          • University of Cincinnati
          • University of Dayton
          • University of Akron

          Users from additional degree granting academic institutions may request to be added to this list per a cost by contacting OSC Help.

          The use of ABAQUS for academic purposes requires validation. In order to obtain validation, please contact OSC Help for further instruction. 

          (link sends e-mail)

          Access for Commerical Users

          Contact OSC Help for getting access to SOFTWARE if you are a commerical user.

          Publisher/Vendor/Repository and License Type

          Dassault Systemes, Commercial

          Usage

          Token Usage

          ABAQUS software usage is monitored though a token-based license manager. This means every time you run a ABAQUS job, tokens are checked out from our pool for your tasks usage. To ensure your job is only started when its required ABAQUS tokens are available it is important to include a software flag within your job script's SBATCH directives.  A minimum of 5 tokens are required per a job, so a 1 node, 1 processor ABAQUS job would need the following SBATCH software flag:  #SBATCH -L abaqus@osc:5 . Jobs requiring more cores will need to request more tokens as calculated with the formula:  M = int(5 x N^0.422) , where N is the total number of cores.  For common requests, you can refer to the following table:

                   Cores            (nodes x ppn each):

          1 2 3 4 6 8 12 16 24 32 48
          Tokens needed: 5 6 7 8 10 12 14 16 19 21 25

          Usage on Owens

          Set-up on Owens

          To load the default version of ABAQUS module, use  module load abaqus . To select a particular software version, use     module load abaqus/version . For example, use  module load abaqus/2016  to load ABAQUS version 2016 on Owens. 

          Using ABAQUS

          Example input data files are available with the ABAQUS release. The  abaqus fetch  utility is used to extract these input files for use. For example, to fetch input file for one of the sample problems including 4 input files, type:

          abaqus fetch job=knee_bolster 

          abaqus fetch job=knee_bolster_ef1 

          abaqus fetch job=knee_bolster_ef2 

          abaqus fetch job=knee_bolster_ef3 

          Also, use the  abaqus help  utility is to list all the abaqus execution procedures.

          Batch Usage on Owens

          When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your ABAQUS analysis to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

          Interactive Batch Session
          For an interactive batch session on Owens, one can run the following command:
          sinteractive -A <project-account> -N 1 -n 28 -t 1:00:00 -L abaqus@osc:5
          
          which gives you 28 cores ( -N 1 -n 28 ) with 1 hour ( -t 1:00:00 ). You may adjust the numbers per your need.
          Non-interactive Batch Job (Serial Run)

          batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice.

          Below is the example batch script ( job.txt ) for a serial run:

          #!/bin/bash
          #SBATCH --time=1:00:00
          #SBATCH --nodes=1 --ntasks-per-node=1
          #SBATCH -L abaqus@osc:5
          #SBATCH --account=<project-account>
          #
          # The following lines set up the ABAQUS environment
          #
          module load abaqus
          #
          cp *.inp $TMPDIR
          cd $TMPDIR
          #
          # Run ABAQUS
          #
          abaqus job=knee_bolster interactive
          #
          # Now, copy data (or move) back once the simulation has completed
          #
          cp * $SLURM_SUBMIT_DIR
          

          In order to run it via the batch system, submit the  job.txt  file with the command:  qsub job.txt 

          NOTE:

          • Make sure to copy all the files needed (input files, restart files, user subroutines, python scripts etc.) from your work directory ( $SLURM_SUBMIT_DIR ) to  $TMPDIR , and copy your results back at the end of your script. Running your job on  $TMPDIR  ensures maximum efficiency.
          • The keyword  interactive  is required in the execution line  abaqus job=knee_bolster interactive  for the following reason: If left off, ABAQUS will background the simulation process. Backgrounding a process in the OSC environment will place it outside of the batch job and it will receive the default 1 hour of CPU time and corresponding default memory limits. The keyword  interactive  in this case simply tells ABAQUS not to return until the simulation has completed.
          • The name of the input file is sometimes omitted in the execution line, which may work fine if you've copied only the input files for one specific model. Although, it is better practice to designate the main input file explicitly by adding  input=<my_input_file_name>.inp  to the execution line so that it looks like  abaqus job=knee_bolster input=<my_input_file_name>.inp interactive .
          • Define  nodes=1  (1<=ppn<=28 for Owens) for a serial run.
          • If ppn>1, add  cpus=<n>  to the execution line, where n=ppn. This is how it should look like:  abaqus job=test input=<my_input_file_name1>.inp cpus=<n> interactive .
          Non-interactive Batch Job (Parallel Run)
          Note: abaqus will not run correctly in parallel with input files in $TMPDIR!  Use the scratch file system.

          Below is the example batch script ( job.txt ) for a parallel run:

          #!/bin/bash 
          #SBATCH --time=1:00:00 
          #SBATCH --nodes=1 --ntasks-per-node=28 --gres=pfsdir
          #SBATCH -L abaqus@osc:27
          #SBATCH --account=<project-account>
          #
          # The following lines set up the ABAQUS environment
          #
          module load abaqus
          #
          # Cope input files to /fs/scratch and run Abaqus there
          #
          cp *.inp $PFSDIR
          cd $PFSDIR
          #
          # Run ABAQUS, note that in this case we have provided the names of the input files explicitly
          #
          abaqus job=test input=<my_input_file_name1>.inp cpus=56 interactive
          #
          # Now, move data back once the simulation has completed
          #
          mv * $SLURM_SUBMIT_DIR
          

          NOTE:

          • If you request a partial node for a serial job (nodes=1:ppn<28), you need to add 'mp_mode=threads' option in order to get the full performance.  
          • Specify  cpus=<n>  in the execution line, where n=nodes*ppn.
          • Everything else is similar to the serial script above.
          • Usage of user-defined material (UMAT) script in Fortran is limited on Owens as following:
            1. abaqus 2017: correctly running on single and multi-nodes
            2. abaqus 6.14 and 2016: correctly running on a single node 

          Further Reading

           

          Supercomputer: 
          Service: 
          Fields of Science: 

          AMBER

          The Assisted Model Building with Energy Refinement (AMBER) package contains many molecular simulation programs targeted at biomolecular systems. A wide variety of modelling techniques are available. It generally scales well on modest numbers of processors, and the GPU enabled CUDA programs are very efficient.

          Availability and Restrictions

          Versions

          AMBER is available on Owens and Pitzer Clusters. The following versions are currently available at OSC (S means serial executables, P means parallel, C means CUDA, i.e., GPU enabled, and M means MIC, i.e., Xeon Phi enabled):

          Version Owens Pitzer Notes
          16 SPC   Default version on Owens prior to 09/04/2018
          18 SPC SPC  
          19 SPC* SPC*  
          20 SPC SPC  
          * Current default version
          *  IMPORTANT NOTE: You need to load correct compiler and MPI modules before you use Amber. In order to find out what modules you need, use module spider amber/{version} .

          You can use module spider amber to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access for Academic Users

          OSC's Amber is available to not-for-profit OSC users; simply contact OSC Help to request the appropriate form for access.

          Access for Commerical Users

          For-profit OSC users must obtain their own Amber license. 

          Publisher/Vendor/Repository and License Type

          University of California, San Francisco, Commercial

          Usage

          Usage on Owens

          Set-up

          To load the default version of AMBER module, use  module load amber . To select a particular software version, use  module load amber/version . For example, use  module load amber/16  to load AMBER version 16. 

          Using AMBER

          A serial Amber program in a short duration run can be executed interactively on the command line, e.g.:

          tleap
          

          Parallel Amber programs must be run in a batch environment with  srun, e.g.:

          srun pmemd.MPI
          

           

          Batch Usage

          When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your AMBER simulation to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

          Interactive Batch Session
          For an interactive batch session, one can run the following command:
          sinteractive -A <project-account> -N 1 -n 28 -t 1:00:00
          
          which gives you one node with 28 cores ( -N 1 -n 28 ), with 1 hour ( -t 1:00:00 ). You may adjust the numbers per your need.
          Non-interactive Batch Job (Serial Run)

          batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Sample batch scripts and Amber input files are available here:

          ~srb/workshops/compchem/amber/
          

          Below is the example batch script ( job.txt ) for a serial run:

          # AMBER Example Batch Script for the Basic Tutorial in the Amber manual
          #!/bin/bash
          #SBATCH --job-name 6pti
          #SBATCH --nodes=1 --ntasks-per-node=28
          #SBATCH --time=0:20:00
          #SBATCH --account=<project-account>
          
          module load amber
          # Use TMPDIR for best performance.
          cd $TMPDIR
          # SLURM_SUBMIT_DIR refers to the directory from which the job was submitted.
          cp -p $SLURM_SUBMIT_DIR/6pti.prmtop .
          cp -p $SLURM_SUBMIT_DIR/6pti.prmcrd .
          # Running minimization for BPTI
          cat << eof > min.in
          # 200 steps of minimization, generalized Born solvent model
          &cntrl
          maxcyc=200, imin=1, cut=12.0, igb=1, ntb=0, ntpr=10,
          /
          eof
          sander -i min.in -o 6pti.min1.out -p 6pti.prmtop -c 6pti.prmcrd -r 6pti.min1.xyz
          cp -p min.in 6pti.min1.out 6pti.min1.xyz $SLURM_SUBMIT_DIR
          

          In order to run it via the batch system, submit the  job.txt  file with the command:  sbatch job.txt

          Usage on Pitzer

          Set-up

          To load the default version of AMBER module, use  module load amber

          Using AMBER

          A serial Amber program in a short duration run can be executed interactively on the command line, e.g.:

          tleap
          

          Parallel Amber programs must be run in a batch environment with mpiexec, e.g.:

          srun pmemd.MPI
          

           

          Batch Usage

          When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your AMBER simulation to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

          Interactive Batch Session
          For an interactive batch session, one can run the following command:
          sinteractive -A <project-account> -N 1 -n 48 -t 1:00:00
          
          which gives you one node with 48 cores ( -N 1 -n 48) with 1 hour ( -t 1:00:00). You may adjust the numbers per your need.
          Non-interactive Batch Job (Serial Run)

          batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Sample batch scripts and Amber input files are available here:

          ~srb/workshops/compchem/amber/
          

          Below is the example batch script ( job.txt ) for a serial run:

          # AMBER Example Batch Script for the Basic Tutorial in the Amber manual 
          #!/bin/bash
          #SBATCH --job-name 6pti #
          SBATCH --nodes=1 --ntasks-per-node=48 
          SBATCH --time=0:20:00
          #SBATCH --account=<project-account>
          
          module load amber
          # Use TMPDIR for best performance.
          cd $TMPDIR
          # SLURM_SUBMIT_DIR refers to the directory from which the job was submitted.
          cp -p $SLURM_SUBMIT_DIR/6pti.prmtop .
          cp -p $SLURM_SUBMIT_DIR/6pti.prmcrd .
          # Running minimization for BPTI
          cat << eof > min.in
          # 200 steps of minimization, generalized Born solvent model
          &cntrl
          maxcyc=200, imin=1, cut=12.0, igb=1, ntb=0, ntpr=10,
          /
          eof
          sander -i min.in -o 6pti.min1.out -p 6pti.prmtop -c 6pti.prmcrd -r 6pti.min1.xyz
          cp -p min.in 6pti.min1.out 6pti.min1.xyz $SLURM_SUBMIT_DIR
          

          In order to run it via the batch system, submit the  job.txt  file with the command:  sbatch job.txt .

          Troubleshooting

          In general, the scientific method should be applied to usage problems.  Users should check all inputs and examine all outputs for the first signs of trouble.  When one cannot find issues with ones inputs, it is often helpful to ask fellow humans, especially labmates, to review the inputs and outputs.  Reproducibility of molecular dynamics simulations is subject to many caveats.  See page 24 of the Amber18 manual for a discussion.

          Further Reading

          Supercomputer: 
          Service: 

          ANSYS

          ANSYS offers a comprehensive software suite that spans the entire range of physics, providing access to virtually any field of engineering simulation that a design process requires. Supports are provided by ANSYS, Inc

          Availability and Restrictions

          Versions

          Version Owens
          17.2 X
          18.1

          X

          19.1 X
          19.2 X
          2019R1 X
          2019R2 X
          2020R1 X*
          2020R2 X
          * Current default version

          OSC has Academic Multiphysics Campus Solution license from Ansys. The license includes most of all the features that Ansys provides. See "Academic Multiphyscis Campus Solution Products" in this table for all available products at OSC.

          Access for Academic Users

          OSC has an "Academic Research " license for ANSYS. This allows for academic use of the software by Ohio faculty and students, with some restrictions. To view current ANSYS node restrictions, please see ANSYS's Terms of Use.

          Use of ANSYS products at OSC for academic purposes requires validation. Please contact OSC Help for further instruction.

          Access for Commerical Users

          Contact OSC Help for getting access to ANSYS if you are a commercial user.

          Publisher/Vendor/Repository and License Type

          Ansys, Inc., Commercial

          Usage

          For more information on how to use each ANSYS product at OSC systems, refer to its documentation page provided at the end of this page.

          Note

          Due to the way our Fluent and ANSYS modules are configured, simultaneously loading multiple of either module will cause a cryptic error. The most common case of this happening is when multiple of a user's jobs are started at the same time and all load the module at once. In order for this error to manifest, the modules have to be loaded at precisely the same time; a rare occurrence, but a probable occurrence over the long term.

          If you encounter this error you are not at fault. Please resubmit the failed job(s).

          If you frequently submit large amounts of Fluent or ANSYS jobs, we recommend you stagger your job submit times to lower the chances of two jobs starting at the same time, and hence loading the module at the same time. Another solution is to establish job dependencies between jobs, so jobs will only start one after another. To do this, you would add the SLURM directive:

          #SBATCH --dependency=after:jobid
          

          To jobs you want to only start after another job has started. You would replace jobid with the job ID of the job to wait for. If you have additional questions, please contact OSC Help.

          Further Reading

          See Also

          Supercomputer: 
          Service: 

          ANSYS Mechanical

          ANSYS Mechanical is a finite element analysis (FEA) tool that enables you to analyze complex product architectures and solve difficult mechanical problems. You can use ANSYS Mechanical to simulate real world behavior of components and sub-systems, and customize it to test design variations quickly and accurately.

          Availability and Restrictions

          ANSYS Mechanical is available on the Owens Cluster. The versions currently available at OSC are:

          Version owens
          17.2  X
          18.1 X
          19.1  X*
          * Current default version

          You can use module spider ansys to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access for Academic Users

          Use of ANSYS for academic purposes requires validation. In order to obtain validation, please contact OSC Help for further instruction.

          Access for Commerical Users

          Contact OSC Help for getting access to ANSYS if you are a commerical user.

          Usage

          Usage on Owens

          Set-up on Owens

          To load the default version of ANSYS module, use  module load ansys . To select a particular software version, use   module load ansys/version . For example, use  module load ansys/17.2   to load ANSYS version 17.2 on Owens. 

          Using ANSYS Mechanical

          Following a successful loading of the ANSYS module, you can access the ANSYS Mechanical commands and utility programs located in your execution path:

          ansys <switch options> <file>
          

          The ANSYS Mechanical command takes a number of Unix-style switches and parameters.

          The -j Switch

          The command accepts a -j switch. It specifies the "job id," which determines the naming of output files. The default is the name of the input file.

          The -d Switch

          The command accepts a -d switch. It specifies the device type. The value can be X11, x11, X11C, x11c, or 3D.

          The -m Switch

          The command accepts a -m switch. It specifies the amount of working storage obtained from the system. The units are megawords.

          The memory requirement for the entire execution will be approximately 5300000 words more than the -m specification. This is calculated for you if you use ansnqs to construct an NQS request.

          The -b [nolist] Switch

          The command accepts a -b switch. It specifies that no user input is expected (batch execution).

          The -s [noread] Switch

          The command accepts a -s switch. By default, the start-up file is read during an interactive session and not read during batch execution. These defaults may be changed with the -s command line argument. The noread option of the -s argument specifies that the start-up file is not to be read, even during an interactive session. Conversely, the -s argument with the -b batch argument forces the reading of the start-up file during batch execution.

          The -g [off] Switch

          The command accepts a -g switch. It specifies that the ANSYS graphical user interface started automatically.

          ANSYS Mechanical parameters

          ANSYS Mechanical parameters may be assigned values on the command. The parameter must be at least two characters long and must be a legal parameter name. The ANSYS Mechanical parameter that is to be assigned a value should be given on the command line with a preceding dash (-), a space immediately after, and the value immediately after the space:

          module load ansys
          ansys -pval1 -10.2 -EEE .1e6
          sets pval1 to -10.2 and EEE to 100000
          

          Batch Usage on Owens

          When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your ANSYS Mechanical analysis to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

          Interactive Batch Session

          Interactive mode is similar to running ANSYS Mechanical on a desktop machine in that the graphical user interface will be sent from OSC and displayed on the local machine. Interactive jobs are run on compute nodes of the cluster, by turning on X11 forwarding. The intention is that users can run ANSYS Mechanical interactively for the purpose of building their model and preparing their input file. Once developed this input file can then be run in no-interactive batch mode.

          To run interactive ANSYS Mechanical, a batch job need to be submitted from the login node, to request necessary compute resources, with X11 forwarding. For example, the following command requests one whole node with 28 cores ( -N 1 -n 28  ), for a walltime of 1 hour (  -t 1:00:00 ), with one ANSYS license:

          sinteractive -N 1 -n 28 -t 1:00:00 -L ansys@osc:1
          

          You may adjust the numbers per your need. This job will queue until resources becomes available. Once the job is started, you're automatically logged in on the compute node; and you can launch ANSYS Mechanical and start the graphic interface with the following commands:

          module load ansys
          ansys -g
          
          Non-interactive Batch Job (Serial Run)

          A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. For a given model, prepare the input file with ANSYS Mechanical commands (named  ansys.in  for example) for the batch run. Below is the example batch script (   job.txt ) for a serial run:

          #!/bin/bash
          #SBATCH --job-name=ansys_test
          #SBATCH --time=1:00:00
          #SBATCH --nodes=1 --ntasks-per-node=1
          #SBATCH -L ansys@osc:1
          
          cd $TMPDIR  
          cp $SLURM_SUBMIT_DIR/ansys.in .    
          module load ansys  
          ansys < ansys.in   
          cp <output files> $SLURM_SUBMIT_DIR
          

          In order to run it via the batch system, submit the  job.txt  file with the command:  qsub job.txt .

          Non-interactive Batch Job (Parallel Run)

          To take advantage of the powerful compute resources at OSC, you may choose to run distributed ANSYS Mechanical for large problems. Multiple nodes and cores can be requested to accelerate the solution time. Note that you'll need to change your batch script slightly for distributed runs.

          Starting from September 15, 2015, a job using HPC tokens (with "ansyspar" flag) should be submitted to Owens clusters due to scheduler issue.

          For distributed ANSYS Mechanical jobs using one node (nodes=1), the number of processors needs to be specified in the command line with options '-dis -np':

          #!/bin/bash
          #SBATCH --job-name=ansys_test
          #SBATCH --time=1:00:00
          #SBATCH --nodes=1 --ntasks-per-node=28
          #SBATCH -L ansys@osc:1,ansyspar@osc:24
          
          ...
          ansys -dis -np 28 < ansys.in  
          ...
          

          Notice that in the script above, the ansys parallel license is requested as well as ansys license in the format of

          #SBATCH -L ansys@osc:1,ansyspar@osc:n
          

          where n=m-4, with m being the total cpus called for this job. This line is necessary when the total cpus called is greater than 4 (m>4), which applies for the parallel example below as well.

          For distributed jobs requesting multiple nodes, you need to specify the number of processors for each node in the command line. This information can be obtained from $PBS_NODEFILE or srun hostname | sort -n. The following shows changes in the batch script if 2 nodes on Owens are requested for a parallel ANSYS Mechanical job:

          #!/bin/bash
          #SBATCH --job-name=ansys_test
          #SBATCH --time=3:00:00
          #SBATCH --nodes=2 --ntasks-per-node=28
          #SBATCH -L ansys@osc:1,ansyspar@osc:52
          
          ...
          export MPI_WORKDIR=$PWD
          machines=`uniq -c ${PBS_NODEFILE} | awk '{print $2 ":" $1}' | paste -s -d ':'`
          ansys -dis -machines $machines < ansys.in  
          ...
          pbsdcp -g '<output files>' $SLURM_SUBMIT_DIR
          

          The 'pbsdcp -g' command in the last line in the script above makes sure that all result files generated by different compute nodes are copied back to the work directory.

          Further Reading

          See Also

          Supercomputer: 
          Service: 

          CFX

          ANSYS CFX (called CFX hereafter) is a computational fluid dynamics (CFD) program for modeling fluid flow and heat transfer in a variety of applications.

          Availability and Restrictions

          CFX is available on the Owens Cluster. The versions currently available at OSC are:

          VERSION owens
          17.2  X
          18.1 X
          19.1 X*
          * Current default version

          You can use module spider fluent  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access for Academic Users

          Use of ANSYS products for academic purposes requires validation. In order to obtain validation, please contact OSC Help for further instruction.

          Currently, there are in total 50 ANSYS CFD base license tokens and 256 HPC tokens for academic users. These base tokens are shared by available ANSYS CFD related projusts (see "Academic Research -> ANSYS Academic Research CFD" in this table for details). These HPC tokens are shared with all ANSYS products we have at OSC. A base license token will allow CFX to use up to 4 cores without any additional tokens. If you want to use more than 4 cores, you will need an additional "HPC" token per core. For instance, a serial CFX job with 1 core will need 1 base license token while a parallel CFX job with 12 cores will need 1 base license token and 8 HPC tokens.

          Access for Commerical Users

          Contact OSC Help for getting access to CFX if you are a commerical user.

          Usage

          Usage on Owens

          Set-up on Owens

          To load the default version, use  module load ansys  . To select a particular software version, use   module load ansys/version . For example, use  module load ansys/17.2   to load CFX version 17.2 on Owens. 

          Batch Usage on owens

          When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your analysis to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

          Interactive Batch Session

          Interactive mode is similar to running CFX on a desktop machine in that the graphical user interface will be sent from OSC and displayed on the local machine. Interactive jobs are run on compute nodes of the cluster, by turning on X11 forwarding. The intention is that users can run CFX interactively for the purpose of building their model and preparing their input file. Once developed this input file can then be run in no-interactive batch mode.

          To run interactive CFX GUI, a batch job need to be submitted from the login node, to request necessary compute resources, with X11 forwarding. Please follwoing the steps below to use CFX GUI interactivly:

          1. Ensure that your SSH client software has X11 forwarding enabled
          2. Connect to Owens system
          3. Request an interactive job. The command below will request one whole node with 28 cores (  -N 1 -n 28  ), for a walltime of one hour ( -t 1:00:00 ), with one ANSYS CFD license (modify as per your own needs):
            sinteractive -N 1 -n 28 -t 1:00:00 -L ansys@osc:1
          4. Once the interactive job has started, run the following commands to setup and start the CFX GUI:

            module load ansys
            cfx5 
            
          Non-interactive Batch Job (Serial Run Using 1 Base Token)

          A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice.

          Below is the example batch script (  job.txt ) for a serial run with an input file test.def ) on Glenn:

          #!/bin/bash
          #SBATCH --job-name=serialjob_cfx
          #SBATCH --time=1:00:00
          #SBATCH --nodes=1 --ntasks-per-node=1
          #SBATCH -L ansys@osc:1
          
          #Set up CFX environment.
          module load ansys
          #Copy CFX files like .def to $TMPDIR and move there to execute the program
          cp test.def $TMPDIR/
          cd $TMPDIR
          #Run CFX in serial with test.def as input file
          cfx5solve -batch -def test.def 
          #Finally, copy files back to your home directory
          cp  * $SLURM_SUBMIT_DIR
          

          In order to run it via the batch system, submit the job.txt  file with the command: qsub job.txt  

          Non-interactive Batch Job (Parallel Execution using HPC token)

          CFX can be run in parallel, but it is very important that you read the documentation in the CFX Manual on the details of how this works.

          In addition to requesting the base license token ( -L ansys@osc:1 ), you need to request copies of the ansyspar license, i.e., HPC tokens ( -L ansys@osc:1,ansyspar@osc:[n] ), where [n] is equal to the number of cores you requested minus 4.

          Parallel jobs have to be submitted on Owens via the batch system. An example of the batch script follows:

          #!/bin/bash
          #SBATCH --job-name=paralleljob_cfx
          #SBATCH --time=10:00:00
          #SBATCH --nodes=2 --ntasks-per-node=28
          #SBATCH -L ansys@osc:1,ansyspar@osc:52
          
          #Set up CFX environment.
          module load ansys
          #Copy CFX files like .def to $TMPDIR and move there to execute the program
          cp test.def $TMPDIR/
          cd $TMPDIR
          #Convert PBS_NODEFILE information into format for CFX host list
          nodes=`cat $PBS_NODEFILE`
          nodes=`echo $nodes | sed -e 's/ /,/g'`
          #Run CFX in parallel with new.def as input file
          #if multiple nodes
          cfx5solve -batch -def test.def  -par-dist $nodes -start-method "Platform MPI Distributed Parallel"
          #if one node
          #cfx5solve -batch -def test.def -par-dist $nodes -start-method "Platform MPI Local Parallel"
          #Finally, copy files back to your home directory
          cp  * $SLURM_SUBMIT_DIR
          

          Further Reading

          Supercomputer: 
          Service: 

          FLUENT

          ANSYS FLUENT (called FLUENT hereafter) is a state-of-the-art computer program for modeling fluid flow and heat transfer in complex geometries.

          Availability and Restrictions

          FLUENT is available on the Owens Cluster. The versions currently available at OSC are:

          Version owens
          17.2  X
          18.1 X
          19.1 X*
          * Current default version

          You can use module spider ansy  for Owens to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access for Academic Users

          Use of ANSYS products for academic purposes requires validation. In order to obtain validation, please contact OSC Help for further instruction.

          Currently, there are in total 50 ANSYS CFD base license tokens and 256 HPC tokens for academic users. These base tokens are shared by available ANSYS CFD related projects (see "Academic Research -> ANSYS Academic Research CFD" in this table for details). These HPC tokens are shared with all ANSYS products we have at OSC. A base license token will allow FLUENT to use up to 4 cores without any additional tokens. If you want to use more than 4 cores, you will need an additional "HPC" token per core. For instance, a serial FLUENT job with 1 core will need 1 base license token while a parallel FLUENT job with 12 cores will need 1 base license token and 8 HPC tokens.

          Access for Commerical Users

          Contact OSC Help for getting access to FLUENT if you are a commercial user.

          Usage

          Usage on Owens

          Set-up on Owens

          To load the default version of FLUENT module, use  module load ansys  . To select a particular software version, use   module load ansys/version . For example, use  module load ansys/17.2   to load FLUENT version 17.2 on Owens. 

          Batch Usage on Owens

          When you log into owens.osc.edu you are actually logged into a Linux box referred to as the login node. To gain access to the multiple processors in the computing environment, you must submit your FLUENT analysis to the batch system for execution. Batch jobs can request multiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

          Interactive Batch Session

          Interactive mode is similar to running FLUENT on a desktop machine in that the graphical user interface will be sent from OSC and displayed on the local machine. Interactive jobs are run on compute nodes of the cluster, by turning on X11 forwarding. The intention is that users can run FLUENT interactively for the purpose of building their model and preparing their input file. Once developed this input file can then be run in non-interactive batch mode.

          To run interactive FLUENT GUI, a batch job need to be submitted from the login node, to request necessary compute resources, with X11 forwarding. Please following the steps below to use FLUENT GUI interactively:

          1. Ensure that your SSH client software has X11 forwarding enabled
          2. Connect to Owens system
          3. Request an interactive job. The command below will request one whole node with 28 cores (  -N 1 -n 28  ), for a walltime of one hour ( -t 1:00:00 ), with one FLUENT license (modify as per your own needs):
            sinteractive -N 1 -n 28 -t 1:00:00 -L ansys@osc:1
          4. Once the interactive job has started, run the following commands to setup and start the FLUENT GUI:

            module load ansys
            fluent 
            
          Non-interactive Batch Job (Serial Run Using 1 Base Token)

          A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice.

          Below is the example batch script (  job.txt ) for a serial run with an input file run.input ) on Owens:

          #!/bin/bash
          #SBATCH --job-name=serial_fluent
          #SBATCH --time=1:00:00
          #SBATCH --nodes=1 --ntasks-per-node=1
          #SBATCH -L ansys@osc:1
          #
          # The following lines set up the FLUENT environment
          #
          module load ansys
          #
          # Copy files to $TMPDIR and move there to execute the program
          #
          cp test_input_file.cas test_input_file.dat run.input $TMPDIR
          cd $TMPDIR
          #
          # Run fluent
          fluent 3d -g < run.input  
          #
          # Where the file 'run.input' contains the commands you would normally
          # type in at the Fluent command prompt.
          # Finally, copy files back to your home directory
          cp *   $SLURM_SUBMIT_DIR 
          

          As an example, your run.input file might contain:

          file/read-case-data test_input_file.cas 
          solve/iterate 100
          file/write-case-data test_result.cas
          file/confirm-overwrite yes    
          exit  
          yes  
          

          In order to run it via the batch system, submit the job.txt  file with the command: qsub job.txt  

          Non-interactive Batch Job (Parallel Execution using HPC token)

          FLUENT can be run in parallel, but it is very important that you read the documentation in the FLUENT Manual on the details of how this works.

          In addition to requesting the FLUENT base license token ( -L ansys@osc:1 ), you need to request copies of the ansyspar license, i.e., HPC tokens ( -L ansys@osc:1,ansyspar@osc:[n] ), where [n] is equal to the number of cores you requested minus 4.

          Parallel jobs have to be submitted to Owens via the batch system. An example of the batch script follows:

          #!/bin/bash
          #SBATCH --job-name=parallel_fluent
          #SBATCH --time=3:00:00
          #SBATCH --nodes=2 --ntasks-per-node=28
          #SBATCH -L ansys@osc:1,ansyspar@osc:52
          set echo on   
          hostname   
          #   
          # The following lines set up the FLUENT environment   
          #   
          module load ansys
          #      
          # Create the config file for socket communication library   
          #   
          # Create list of nodes to launch job on   
          rm -f pnodes   
          cat  $PBS_NODEFILE | sort > pnodes   
          export ncpus=`cat pnodes | wc -l`   
          #   
          #   Run fluent   
          fluent 3d -t$ncpus -pinfiniband.ofed -cnf=pnodes -g < run.input 
          

          Further Reading

          See Also

          Supercomputer: 
          Service: 

          Workbench Platform

          ANSYS Workbench platform is the backbone for delivering a comprehensive and integrated simulation system to users. See ANSYS Workbench platform for more information. 

          Availability and Restrictions

          ANSYS Workbench is available on Owens Cluster. The versions currently available at OSC are:

          Version owens
          17.2 SF  X
            CFD  X
          18.1 SF X
            CFD X
          19.1   X*
          * Current default version

          Note:

          • SF: Structural-Fluid dynamics related applications. See "Academic Research -> ANSYS Academic Research Mechanical and CFD" in this table for all available products
          • CFD: CFD related applications. See "Academic Research -> ANSYS Academic Research CFD" in this table for all available products

          You can use module spider ansys  to view available modules for a given machine if you want to use structural-fluid dynamics related applications or module spider fluent   to view available modules for a given machine if you want to use CFD related applications. Feel free to contact OSC Help if you need other versions for your work.

          Access for Academic Users

          Use of ANSYS products for academic purposes requires validation. In order to obtain validation, please contact OSC Help for further instruction.

          Access for Commerical Users

          Contact OSC Help for getting access to ANSYS if you are a commerical user.

          Usage

          Usage on Owens

          Set-up for Structural-Fluid dynamics related applications

          To load the default version , use  module load ansys . To select a particular software version, use   module load ansys/version . For example, use  module load ansys/17.2   to load version 17.2 on Owens. After the module is loaded, use the following command to open Workbench GUI:

          runwb2
          

          Set-up for CFD related applications

          To load the default version , use  module load ansys  . To select a particular software version, use   module load ansys/version   . For example, use  module load ansys/17.2   to load version 17.2 on Owens. After the module is loaded, use the following command to open Workbench GUI:

          runwb2
          

          Further Reading

          See Also

          Supercomputer: 
          Service: 

          ARM HPC tools

          ARM HPC tools analyze how HPC software runs. It consists of three applications, ARM DDT, ARM Performance Reports and ARM MAP: 

          • ARM DDT: graphical debugger for HPC applications.
          • ARM MAP: HPC application profiler with easy-to-use GUI environment.
          • ARM Performance Reports: simple tool to generate a single-page HTML or plain text report that presents overall performance characteristics of HPC applications.

           

          NOTE: Because ARM has aquired Allinea, all Allinea module files have been renamed accordingly. Allinea modules are still available and have same functionality as new ARM modules.

          Availability & Restrictions

          Versions

          The following versions of ARM HPC tools are available on OSC clusters:

          Version Owens Pitzer
          6.0.6 X  
          6.1.1    
          7.0

          X

           
          7.1 X  
          18.2.1 X X
          19.0.1 X X
          19.1.2 X X
          20.0.3 X* X*
          * Current default version

          You can use module spider arm to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          ARM DDT, MAP and Performance Reports are available to all OSC users.

          Publisher/Vendor/Repository and License Type

          ARM, Commercial

          Usage

          ARM DDT

          ARM DDT is a debugger for HPC software that automatically alerts users of memory bugs and divergent behavior. For more features and benefits, visit ARM HPC tools and libraries - DDT.

          For usage instructions and more iformation, read ARM DDT.

          ARM MAP

          ARM MAP produces a detailed profile of HPC software. Unlike ARM Performance Reports, you must have the source code to run ARM MAP because its analysis details the software line-by-line. For more features and benefits, visit ARM HPC tools and libraries - MAP

          For usage instructions and more information, read ARM MAP.

          ARM Performance Reports

          ARM Performance Reports analyzes and documents information on CPU, MPI, I/O, and Memory performance characteristics of HPC software, even third party code, to aid understanding about the overall performance. Although it should not be used all the time, ARM Performance Reports is recommended to OSC users as a viable option to analyze how an HPC application runs. View an example report to navigate the format of a typical report. For more example reports, features and benefits, visit ARM HPC tools and libraries - Performance Reports.

          For usage instructions and more information, read ARM Performance Reports.

          Troubleshooting

          Using ARM software with MVAPICH2

          This note from ARM's Getting Started Guide applies to both perf-report and MAP:

          Some MPIs, most notably MVAPICH, are not yet supported by ARM's Express Launch mode
          (in which you can just put “perf-report” in front of an existing mpirun/mpiexec line). These can
          still be measured using the Compatibility Launch mode.

          Instead of this Express Launch command:

          perf-report mpiexec <mpi args> <program> <program args> # BAD
          

          Use the compatibility launch version instead:

          perf-report -n <num procs> --mpiargs="<mpi args>" <program> <program args>
          

          Further Reading

          See Also

          Documentation Attachment: 
          Supercomputer: 
          Service: 
          Fields of Science: 

          ARM Performance Reports

          ARM Performance Reports is a simple tool used to generate a single-page HTML or plain text report that presents the overall performance characteristics of HPC applications. It supports pthreads, OpenMP, or MPI code on CPU, GPU, and MIC based architectures.

          Availability and Restrictions

          Versions

          The versions currently available at OSC are:

          Version Owens Pitzer
          6.0.6 X  
          6.1.1    
          7.0

          X

           
          7.1 X  
          18.2.1 X X
          19.0.1 X X
          19.1.2 X X
          20.0.3 X* X*
          * Current default version

          You can use module spider arm-pr to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          ARM Performance Reports is available to all OSC users. We have 64 seats with 64 HPC tokens. Users can monitor the license status here.

          Publisher/Vendor and License Type

          ARM, Commercial

          Usage

          Set-up

          To load the module for the ARM Performance Reports default version, use module load arm-pr. To select a particular software version, use module load arm-pr/version. For example, use module load arm-pr/6.0 to load ARM Performance Reports version 6.0, provided the version is available on the OSC cluster in use.

          Using ARM Performance Reports

          You can use your regular executables to generate performance reports. The program can be used to analyze third-party code as well as code you develop yourself. Performance reports are normally generated in a batch job.

          To generate a performance report for an MPI program:

          module load arm-pr
          perf-report -np <num procs> --mpiargs="<mpi args>" <program> <program args>
          

          where <num procs> is the number of MPI processes to use, <mpi args> represents arguments to be passed to mpiexec (other than -n or -np), <program> is the executable to be run and <program args> represents arguments passed to your program.

          For example, if you normally run your program with mpiexec -n 12 wave_c, you would use

          perf-report -np 12 wave_c
          

          To generate a performance report for a non-MPI program:

          module load arm-pr
          perf-report --no-mpi <program> <program args>
          

          The performance report is created in both html and plain text formats. The file names are based on the executable name, number of processes, date and time, for example,  wave_c_12p_2016-02-05_12-46.html. To open the report in html format use

          firefox wave_c_12p_2016-02-05_12-46.html
          

          For more details, download the ARM Performance Reports User Guide.

          Performance Reports with GPU

          ARM Performance Reports can be used for CUDA codes. If you have an executable compiled with the CUDA library, you can launch ARM Performance Reports with

          perf-report {executable}
          

          For more information, please read the section 6.10 of the ARM Performance Reports User Guide.

          Further Reading

          See Also

          Documentation Attachment: 
          Supercomputer: 
          Service: 

          ARM MAP

          ARM MAP is a full scale profiler for HPC programs. We recommend using ARM MAP after reviewing reports from ARM Performance Reports. MAP supports pthreads, OpenMP, and MPI software on CPU, GPU, and MIC based architectures.

          Availability & Restrictions

          Versions

          The ARM MAP versions currently available at OSC are:

          Version Ruby Owens Pitzer
          5.0.1 X    
          5.1 X    
          6.0 X    
          6.0.1 X    
          6.0.6   X  
          6.1.1 X    
          7.0 X

          X

           
          7.1   X  
          18.2.1 X* X X
          19.0.1   X X
          19.1.2   X X
          20.0.3   X* X*
          * Current default version

          You can use module spider arm-map to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          ARM MAP is available to all OSC users. We have 64 seats with 80 HPC tokens. Users can monitor the ARM License Server Status.

          Publisher/Vendor and License Type

          ARM, Commercial

          Usage

          Set-up

          To load the default version of the ARM MAP module, use module load arm-map. To select a particular software version, use module load arm-map/version. For example, use module load arm-map/6.0 to load ARM MAP version 6.0, provided the version is available on the cluster in use. 

          Using ARM MAP

          Profiling HPC software with ARM MAP typically involves three steps: 

          1. Prepare the executable for profiling.

          Regular executables can be profiled with ARM MAP, but source code line detail will not be available. You need executables with debugging information to view source code line detail: re-compile your code with a -g  option added among the other appropriate compiler options. For example:

          mpicc wave.c -o wave -g -O3
          

          This executable built with the debug flag can be used for ARM Performance Reports as well.

          Note: The -g flag turns off all optimizations by default. For profiling your code you should use the same optimizations as your regular executable, so explicitly include the -On flag, where n is your normal level of optimization, typically -O2 or -O3, as well as any other compiler optimization options.

          2. Run your code to produce the profile data file (.map file).

          Profiles are normally generated in a batch job.  To generate a MAP profile for an MPI program:

          module load arm-map
          map --profile -np <num proc> --mpiargs="<mpi args>" <program> <program args>

          where <num procs> is the number of MPI processes to use, <mpi args> represents arguments to be passed to mpiexec (other than -n or -np), <program> is the executable to be run and <program args> represents arguments passed to your program.

          For example, if you normally run your program with mpiexec -n 12 wave_c, you would use

          map --profile -np 12 wave_c

          To profile a non-MPI program:

          module load arm-map
          map --profile --no-mpi <program> <program args>

          The profile data is saved in a .map file in your current directory.

          As a result of this step, a .map file that is the profile data file is created in your current directory. The file name is based on the executable name, number of processes, date and time, for example, wave_c_12p_2016-02-05_12-46.map.

          For more details on using ARM MAP, refer to the ARM Forge User Guide.

          3. Analyze the profile data file using either the ARM local client or the MAP GUI.

          You can open the profile data file using a client running on your local desktop computer. For client installation and usage instructions, please refer to the section: Client Download and Setup. This option typically offers the best performance.

          Alternatively, you can run MAP in interactive mode, which launches the graphical user interface (GUI).  For example:

          map wave_c_12p_2016-02-05_12-46.map

          For the GUI application, one should use an OnDemand VDI (Virtual Desktop Interface) or have X11 forwarding enabled (see Setting up X Windows). Note that X11 forwarding can be distractingly slow for interactive applications.

          MAP with GPU

          ARM MAP can be used for CUDA codes. If you have an executable compiled with the CUDA library, you can launch ARM MAP with

          map {executable}
          

          For more information, please read the Chapter 15 of the ARM Forge User Guide.

          Client Download and Setup

          1. Download the client.

          To download the client, go to the ARM website and choose the appropriate ARM Forge remote client download for Windows, Mac, or Linux. For Windows and Mac, just double click on the downloaded file and allow the installer to run. For Linux, extract the tar file using the command tar -xf file_name and run the installer in the extracted file directory with ./installer. Please contact OSC Help, if you have any issues on downloading the client.

          2. Configure the client.

          After installation, you can configure the client as follows:

          • Open the client program. For Windows or Mac, just click the desktop icon or navigate to the application through its file path. For Linux use the command {arm-forge-path}/bin/map.

          • Once the program is launched, select ARM MAP in the left column.
          • In the Remote Launch drop down menu, select "Configure...".
          • Click Add to create a new profile for your login.
          • In the Host Name section, type your ssh connection. For example: "username@ruby.osc.edu".
          • For Remote Installation Directory, type /usr/local/arm/forge-{version}, specifying the ARM Forge version number that created the data profile file you are attempting to view. For example, /usr/local/arm/forge-7.0 for ARM Forge version 7.0.
          • You can test your login information by clicking Test Remote Launch. It will ask your password. Use the same password for the cluster login.
          • Close the Configure window. You will see a new option under the Remote Launch drop down menu for the host name you entered. Select your profile and login with your password. 
          • If the login was successful, then you should see License Serial:XXX in the bottom left corner of the window.

          This login configuration is needed only for the first time of use. In subsequent times, you can just select your profile.

          3. Open the profile data file.

          After login, click on LOAD PROFILE DATA FILE. This opens a file browser of your home directory on the OSC cluster you logged onto. Go to the directory that contains the .map file and select it. This will open the file and allow you to navigate the source code line-by-line and investigate the performance characteristics. 

          A license is not required to simply open the client, so it is possible to skip 2. Configure the client, if you download the profile data file to your desktop. You can then open it by just selecting LOAD PROFILE DATA FILE and navigating through a file browser on your local system.

          Note that the client is ARM Forge, a client that contains ARM MAP and ARM DDT. ARM DDT is a debugger, and OSC has license only for ARM MAP. If you need a debugger, you can use Totalview instead.

          Further Reading

          See Also

          Documentation Attachment: 
          Supercomputer: 
          Service: 
          Fields of Science: 

          ARM DDT

          Arm DDT is a graphical debugger for HPC applications. It supports pthreads, OpenMP, or MPI code on CPU, GPU, and MIC based architectures.

          Availability & Restrictions

          Versions

          The Arm DDT versions currently available at OSC are:

          Version Owens Pitzer
          6.0.6 X  
          6.1.1    
          7.0

          X

           
          7.1 X  
          18.2.1 X X
          19.0.1 X X
          19.1.2 X X
          20.0.3 X* X*
          * Current default version

          You can use module spider arm-ddt to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          Arm DDT is available to all OSC users. We have 64 seats with 80 HPC tokens. Users can monitor the Arm License Server Status.

          Publisher/Vendor and License Type

          ARM, Commercial

          Usage

          Set-up

          To load the module for the Arm DDT default version, use module load arm-ddt. To select a particular software version, use module load arm-ddt/version. For example, use module load arm-ddt/7.0 to load Arm DDT version 7.0, provided the version is available on the OSC cluster in use.

          Using Arm DDT

          DDT debugs executables to generate DDT reports. The program can be used to debug third-party code as well as code you develop yourself. DDT reports are normally generated in a batch job.

          To generate a DDT report for an MPI program:

          module load arm-ddt
          ddt --offline -np <num procs> --mpiargs="<mpi args>" <program> <program args>
          

          where <num procs> is the number of MPI processes to use, <mpi args> represents arguments to be passed to mpiexec (other than -n or -np), <program> is the executable to be run and <program args> represents arguments passed to your program.

          For example, if you normally run your program with mpiexec -n 12 wave_c, you would use

          ddt --offline -np 12 wave_c
          

          To debug a non-MPI program:

          module load arm-ddt
          ddt --offline --no-mpi <program> <program args>
          

          The DDT report is created in html format. The file names are based on the executable name, number of processes, date and time, for example, wave_c_12p_2016-02-05_12-46.html. To open the report use

          firefox wave_c_12p_2016-02-05_12-46.html
          

          Using the Arm DDT GUI

          To debug with the DDT GUI remove the --offline option. For example, to debug the MPI program above, use

          ddt -np 12 wave_c

          For a non-MPI program:

          ddt --no-mpi <program> <program args>

          This will open the DDT GUI, enabling interactive debugging options.

          For the GUI application, one should use an OnDemand VDI (Virtual Desktop Interface) or have X11 forwarding enabled (see Setting up X Windows). Note that X11 forwarding can be distractingly slow for interactive applications.

          For more details, see the Arm DDT developer page.

          DDT with GPU

          DDT can be used for CUDA codes. If you have an executable compiled with the CUDA library, you can launch Arm Performance Reports with

          ddt {executable}
          

          For more information, please read the chapter 14 of the Arm Forge User Guide.

          Supercomputer: 

          Altair HyperWorks

          HyperWorks is a high-performance, comprehensive toolbox of CAE software for engineering design and simulation.

          Availability & Restrictions

          Versions

          The following version of Altair Hyperworks can be found for the following environments:

          Version Owens Statewide
          10   X
          11   X
          12   X
          13 X X
          14   X
          2017.1 X X
          2019.2 X* X
          2020.0 X X
          * Current Default Version

          You can use module spider hyperworks to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          HyperWorks is available to all academic clients. Please contact OSC Help to request the appropriate form for access.

          Publisher/Vendor/Repository and License Type

          Altair Engineering, Commercial (state-wide)

          Usage

          Using HyperWorks through OSC installation

          To use HyperWorks on the OSC clusters, first ensure that X11 forwarding is enabled as the HyperWorks workbench is a graphical application. Then, load the hyperworks module:

          module load hyperworks
          

          The HyperWorks GUI can be launched then with the following command:

          hw
          

          The Hypermesh GUI can be launched then with the following command:

          hm

          State-wide access for HyperWorks

          For information on downloading and installing a local copy through the state-wide license, follow the steps below. 

          NOTE: To run Altair HyperWorks, your computer must have access to the internet. The software contacts the license server at OSC to check out a license when it starts and periodically during execution. The amount of data transferred is small, so network connections over modems are acceptable.

           

          Usage of HyperWorks on a local machine using the statewide license will vary from installation to installation.

          1. Go to http://www.altairhyperworks.com/ 

          2. Click on "Login" in the upper right hand corner of the page. 

          3. If you have already registered with the Altair web site, enter the e-mail address that you registered with and your password and skip to step #5.

          4. If you have not registered yet, click the link that says " Sign up for Altair Connect". You will be prompted for some contact information and an e-mail address which will be your unique identifier.

            • IMPORTANT: The e-mail address you give must be from your academic institution. Under the statewide license agreement, registration from Ohio universities is allowed on the Altair web site. Trying to log in with a yahoo or hotmail e-mail account will not work. If you enter your university e-mail and the system will not register you, please contact OSChelp at oschelp@osc.edu.

          5. Once you have logged in, click on "SUPPORT" and then "SOFTWARE DOWNLOADS"

          6. In addition to downloading the software, download the "Installation Guide and Release Notes" for instructions on how to install the software.

            • IMPORTANT: If you have any questions or problems, please contact OSChelp at oschelp@osc.edu, rather than HyperWorks support. The software agreements outlines that problems should first be sent to OSC. If the OSC support line cannot answer or resolve the question, they have the ability to raise the problem to Altair support.

          7. Please contact OSC Help for further instruction and license server information. In order to be added to the allowed list for the state-wide software access, we will need your IP address/range of machine that will be running this software. 

          8. You need to set an environment variable (ALTAIR_LICENSE_PATH) on your local machine to point at our license server (7790@license6.osc.edu). See this link for instructions if necessary.

          Further Reading

          For more information about HyperWorks, see the following:

          See Also

          Supercomputer: 
          Service: 
          Fields of Science: 

          BLAS

          The BLAS (Basic Linear Algebra Subprograms) are routines that provide standard building blocks for performing basic vector and matrix operations.

          Availability and Restrictions

          Access

          A highly optimized implementation of the BLAS is available on all OSC clusters as part of the Intel Math Kernel Library (MKL). We recommend that you use MKL rather than building the BLAS for yourself. MKL is available to all OSC users.

          Usage

          See OSC's MKL software page for usage information. Note that there is no library named libblas.a or libblas.so. The flag "-lblas" on your link line will not work. You should modify your makefile or build script to link to the MKL libraries instead.

          Further Reading

          Service: 
          Technologies: 
          Fields of Science: 

          BLAST

          The BLAST programs are widely used tools for searching DNA and protein databases for sequence similarity to identify homologs to a query sequence. While often referred to as just "BLAST", this can really be thought of as a set of programs: blastp, blastn, blastx, tblastn, and tblastx.

          Availability & Restrictions

          Versions

          The following versions of BLAST are available on OSC systems: 

          Version Owens Pitzer
          2.4.0+ X  
          2.8.0+   X
          2.8.1+ X  
          2.10.0+ X* X*
          * Current Default Version

          You can use module spider blast to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          If you need to use blastx, you will need to load one of the C++ implimenations modules of blast (any version with a "+").

          Access

          BLAST is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          National Institutes of Health, Open source

          Usage

          Set-up

          To load BLAST, type the following into the command line:

          module load blast

          Then create a resource file .ncbirc, and put it under your home directory.

          Using BLAST

          The five flavors of BLAST mentioned above perform the following tasks:

          • blastp: compares an amino acid query sequence against a protein sequence database

          • blastn: compares a nucleotide query sequence against a nucleotide sequence database

          • blastx: compares the six-frame conceptual translation products of a nucleotide query sequence (both strands) against a protein sequence database

          • tblastn: compares a protein query sequence against a nucleotide sequence database dynamically translated in all six reading frames (both strands).

          • tblastx: compares the six-frame translations of a nucleotide query sequence against the six-frame translations of a nucleotide sequence database. (Due to the nature of tblastx, gapped alignments are not available with this option)

          NCBI BLAST Database

          Information on the NCBI BLAST database can be found here. https://www.osc.edu/resources/available_software/scientific_database_list/blast_database 

          We provide local access to nt and refseq_protein databases. You can access the database by loading desired blast-database modules. If you need other databases, please send a request email to OSC Help .

          Batch Usage

          A sample batch script on Owens and Pitzer is below:

          #!/bin/bash
          ## --ntasks-per-node can be increased upto 48 on Pitzer
          #SBATCH --nodes=1 --ntasks-per-node=28 
          #SBATCH --time=00:10:00
          #SBATCH --job-name Blast
          #SBATCH --account=<project-account>
          
          module load blast
          module load blast-database/2018-08
          
          cp 100.fasta $TMPDIR
          cd $TMPDIR
          
          tblastn -db nt -query 100.fasta -num_threads 16 -out 100_tblastn.out
          
          cp 100_tblastn.out $SLURM_SUBMIT_DIR
          

          Further Reading

          Supercomputer: 
          Service: 
          Fields of Science: 

          BWA

          BWA is a software package for mapping low-divergent sequences against a large reference genome, such as the human genome. It consists of three algorithms: BWA-backtrack, BWA-SW and BWA-MEM.

          Availability and Restrictions

          Versions

          The following versions of BWA are available on OSC clusters:

          Version Owens Pitzer
          0.7.17-r1198 X* X*
          * Current default version

          You can use module spider bwa to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          BWA is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Li H. and Durbin R., Open source

          Usage

          Usage on Owens

          Set-up

          To configure your environment for use of BWA, run the following command: module load bwa. The default version will be loaded. To select a particular BWA version, use module load bwa/version. For example, use module load bwa/0.7.13 to load BWA 0.7.13.

          Usage on Pitzer

          Set-up

          To configure your environment for use of BWA, run the following command: module load bwa. The default version will be loaded.

          Further Reading

          Supercomputer: 
          Service: 
          Fields of Science: 

          BamTools

          BamTools provides both a programmer's API and an end-user's toolkit for handling BAM files.

          Availability and Restrictions

          Versions

          The following versions of BamTools are available on OSC clusters:

          Version Owens Pitzer
          2.2.2  X*  
          2.3.0   X*
          * Current default version

          You can use module spider bamtools to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          BamTools is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Derek Barnett, Erik Garrison, Gabor Marth, and Michael Stromberg/ Open Source

          Usage

          Usage on Owens

          Set-up

          To configure your environment for use of BamTools, run the following command: module load bamtools. The default version will be loaded. To select a particular BamTools version, use module load bamtools/version. For example, use module load bamtools/2.2.2 to load BamTools 2.2.2.

          Usage on Pitzer

          Set-up

          To configure your environment for use of BamTools, run the following command: module load bamtools. The default version will be loaded.

          Further Reading

          Supercomputer: 
          Service: 
          Fields of Science: 

          Bismark

          Bismark is a program to map bisulfite treated sequencing reads to a genome of interest and perform methylation calls in a single step.

          Availability and Restrictions

          Versions

          The following versions of bedtools are available on OSC clusters:

          Version Owens Pitzer
          0.22.1 X* X*
          0.22.3 X X
          * Current default version

          You can use module spider bismark to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          Bismark is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Babraham Bioinformatics, GNU GPL v3

          Usage

          Usage on Owens

          Set-up

          To configure your environment for use of metilene, run the following command: module load bismark. The default version will be loaded. To select a particular Bismark version, use module load bismark/version. For example, use module load bismark/0.22.1 to load Bismark 0.22.1.

          Usage on Pitzer

          Set-up

          To configure your environment for use of metilene, run the following command: module load bismark. The default version will be loaded. To select a particular Bismark version, use module load bismark/version. For example, use module load bismark/0.22.1 to load Bismark 0.22.1.

          Further Reading

          Supercomputer: 
          Service: 
          Fields of Science: 

          Blender

          Blender is the free and open source 3D creation suite. It supports the entirety of the 3D pipeline—modeling, rigging, animation, simulation, rendering, compositing and motion tracking, even video editing and game creation.

          Availability and Restrictions

          Versions

          The following versions of Blender are available on OSC systems: 

          Version Owens Pitzer
          2.79 X*  
          2.91 X X*
          * Current default version

          You can use module spider blender to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          Blender is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Blender Foundation, Open source

          Usage

          Set-up

          On Owens-Desktop 'vis' or 'any' node type, run the following command:

          module load blender
          

          Using Blender

          To run hardware-rendering version of blender, connect to OSC OnDemand and luanch a virtual desktop, either a Virtual Desktop Interface (VDI) or an Interactive HPC 'vis' type Desktop, and in desktop open a terminal and run blender with VirtualGL

          module load virtualgl
          vglrun blender
          

          You can also run software-rendering version of blender on any type Desktop: 

          blender-softwaregl

          Further Reading

           

          Tag: 
          Supercomputer: 
          Service: 
          Fields of Science: 

          Boost

          Boost is a set of C++ libraries that provide helpful data structures and numerous support functions in a wide range of aspects of programming, such as, image processing, gpu programming, concurrent programming, along with many algorithms.  Boost is portable and performs well on a wide variety of platforms.

          Availability & Restrictions

          Versions

          The following version of Boost are available on OSC systems:

          Version Owens Pitzer Notes
          1.53.0 System Install   No Module Needed
          1.56.0      
          1.63.0 X(GI)    
          1.64.0 X(GI)    
          1.67.0 X(GI) X(GI)  
          1.72.0 X(GI)* X(GI)*  
          * Current default version; G = available with gnu; I = available with intel

          You can use module spider boost to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          Boost is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Beman Dawes, David Abrahams, Rene Rivera/ Open source

          Usage

          Usage on Owens

          Set-up

          Initalizing the system for use of the Boost library is independent of the compiler you are using.  To load the boost module run the following command:

          module load boost

          Building With Boost

          The following environment variables are setup when the Boost library is loaded:

          VARIABLE USE
          $BOOST_CFLAGS Use during your compilation step for C++ programs.
          $BOOST_LIBS Use during your link step.

           

          Below is a set of example commands used to build and run a file called  example2.cpp. First copy the example2.cpp and jayne.txt from Oakley into your home directory with the following commands:

          cp /usr/local/src/boost/boost-1_56_0/test.osc/example2.cpp ~
          cp /usr/local/src/boost/boost-1_56_0/test.osc/jayne.txt ~
          
          Then compile and test the program with the folllowing commands:
          g++ example2.cpp -o boostTest -lboost_regex
          ./boostTest < jayne.txt
          

          Usage on Pitzer

          Set-up

          Initalizing the system for use of the Boost library is independent of the compiler you are using.  To load the boost module run the following command:

          module load boost

          Building With Boost

          The following environment variables are setup when the Boost library is loaded:

          VARIABLE USE
          $BOOST_CFLAGS Use during your compilation step for C++ programs.
          $BOOST_LIBS Use during your link step.

           

          Below is a set of example commands used to build and run a file called  example2.cpp. First copy the example2.cpp and jayne.txt from Oakley into your home directory with the following commands:

          cp /usr/local/src/boost/boost-1_56_0/test.osc/example2.cpp ~
          cp /usr/local/src/boost/boost-1_56_0/test.osc/jayne.txt ~
          
          Then compile and test the program with the folllowing commands:
          g++ example2.cpp -o boostTest -lboost_regex
          ./boostTest < jayne.txt

          Further Reading

           

          Supercomputer: 
          Service: 
          Fields of Science: 

          Bowtie1

          Bowtie1 is an ultrafast, memory-efficient short read aligner. It aligns short DNA sequences (reads) to the human genome at a rate of over 25 million 35-bp reads per hour. Bowtie indexes the genome with a Burrows-Wheeler index to keep its memory footprint small: typically about 2.2 GB for the human genome (2.9 GB for paired-end).

          Availability and Restrictions

          Versions

          The following versions of Bowtie1 are available on OSC clusters:

          Version Owens Pitzer
          1.1.2 X*  
          1.2.2   X*
          * Current default version

          You can use module spider bowtie1 to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          Bowtie1 is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Ben Langmead et al., Open source (Artistic 2.0)

          Usage

          Usage on Owens

          Set-up

          To configure your environment for use of Bowtie1, run the following command:  module load bowtie1. The default version will be loaded. To select a particular Bowtie1 version, use  module load bowtie/version. For example, use  module load bowtie1/1.1.2 to load Bowtie1 1.1.2.

          Usage on Pitzer

          Set-up

          To configure your environment for use of Bowtie1, run the following command:  module load bowtie1. The default version will be loaded. 

          Further Reading

          Supercomputer: 
          Service: 
          Fields of Science: 

          Bowtie2

          Bowtie2 is an ultrafast and memory-efficient tool for aligning sequencing reads to long reference sequences. It is particularly good at aligning reads of about 50 up to 100s or 1,000s of characters, and particularly good at aligning to relatively long (e.g. mammalian) genomes. Bowtie 2 indexes the genome with an FM Index to keep its memory footprint small: for the human genome, its memory footprint is typically around 3.2 GB. Bowtie 2 supports gapped, local, and paired-end alignment modes.

          Please note that bowtie (and tophat) CANNOT run in parallel, that is, on multiple nodes.  Submitting multi-node jobs will only waste resources.  In addition you must explicitly include the '-p' option to use multiple threads on a single node.

          Availability and Restrictions

          Versions

          The following versions of Bowtie2 are available on OSC clusters:

          Version Owens Pitzer Note
          2.2.9 X    
          2.3.4.3   X  
          2.4.1 X* X* Python 3 rqeuired for all python scripts
          * Current default version

          You can use module spider bowtie2 to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          Bowtie2 is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Ben Langmead et al., Open source

          Usage

          Usage on Owens

          Set-up

          To configure your environment for use of Bowtie2, run the following command: module load bowtie2. The default version will be loaded. To select a particular Bowtie2 version, use module load bowtie2/version. For example, use  module load bowtie2/2.2.9 to load Bowtie2 2.2.9.

          Usage on Pitzer

          Set-up

          To configure your environment for use of Bowtie2, run the following command: module load bowtie2. The default version will be loaded.

          Further Reading

          Supercomputer: 
          Service: 
          Fields of Science: 

          CMake

          CMake is a family of compilation tools that can be used to build, test and package software.

          Availability and Restrictions

          Versions

          The current versions of CMake available at OSC are:

          Version Owens Pitzer
          2.8.12.2 X#  
          3.1.1    
          3.6.1 X  
          3.7.2 X  
          3.11.4 X X
          3.16.5 X X
          3.17.2 X* X*
          * Current default version; # System version

          You can use  module spider cmake  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          CMake is available to all OSC users.

          Publisher/Vendor/Repository and License Type

          Aaron C. Meadows et al., Open source

          Usage

          Usage on Owens

          Set-up

          To configure your environment for use of CMake, run the following command:  module load cmake . The default version will be loaded. To select a particular CMake version, use  module load cmake/version . For example, use   module load cmake/3.6.1  to load CMake 3.6.1. For the system version of CMake, you can use it without module load

          Usage on Pitzer

          Set-up

          To configure your environment for use of CMake, run the following command:  module load cmake . The default version will be loaded.

          Further Reading

          For more information, visit the CMake homepage.

          Supercomputer: 

          COMSOL

          COMSOL Multiphysics (formerly FEMLAB) is a finite element analysis and solver software package for various physics and engineering applications, especially coupled phenomena, or multiphysics. owned and supported by COMSOL, Inc.

          Availability and Restrictions

          Versions

          COMSOL is available on the Owens clusters. The versions currently available at OSC are:

          Version Owens
          52a X
          53a X
          5.4 X
          5.5 X*
          * Current default version

          You can use module spider comsol  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access for Academic Users

          COMSOL is for academic use, available to all Ohio State University users. OSC does not provide COMSOL licenses for academic use to students and faculty outside of Ohio State University due to licensing restrictions. If you or your institution have a network COMSOL license server, you may be able to use it on OSC. For connections to your license server from OSC, please read this document. If you need further help, please contact OSC Help.

          To use COMSOL you will have to be added to the license server.  Please contact OSC Help to be added.

          Access for Commerical Users

          Contact OSC Help for getting access to COMSOL if you are a commercial user. 

          Publisher/Vendor/Repository and License Type

          Comsol Inc., Commercial

          Usage

          Usage on Owens

          Set-up

          To load the default version of COMSOL module, use  module load comsol . To select a particular software version, use   module load comsol/version . For example, use  module load comsol/52a  to load COMSOL version 5.2a. 

          Batch Usage

          When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your analysis to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

          Interactive Batch Session
          For an interactive batch session, one can run the following command:
          sinteractive -A <project-account> -N 1 -n 28 -t 1:00:00 -L comsolscript@osc:1
          
          which gives you 28 cores ( -N 1 -n 28 ) with 1 hour ( -t 1:00:00 ). You may adjust the numbers per your need.
          Non-interactive Batch Job (Serial Run)

          Assume that you have had a comsol script file  mycomsol.m  in your working direcory ( $SLURM_SUBMIT_DIR ). Below is the example batch script ( job.txt ) for a serial run: 

          #!/bin/bash
          #SBATCH --time=1:00:00
          #SBATCH --nodes=1 --ntasks-per-node=1
          #SBATCH -L comsolscript@osc:1
          #SBATCH --account=<project-account>
          #
          # The following lines set up the COMSOL environment
          #
          module load comsol
          #
          cp *.m $TMPDIR
          cd $TMPDIR
          #
          # Run COMSOL
          #
          comsol batch mycomsol
          #
          # Now, copy data (or move) back once the simulation has completed
          #
          cp * $SLURM_SUBMIT_DIR
          
          Non-interactive Batch Job (Parallel Run for COMSOL 4.3 and Later)

          As of version 4.3, it is not necessary to start up MPD before launching a COMSOL job. Below is the example batch script ( job.txt ) for a parallel run using COMSOL 4.3 or later versions:

          #!/bin/bash
          #SBATCH --time=1:00:00
          #SBATCH --nodes=2 --ntasks-per-node=28
          #SBATCH -L comsolscript@osc:1
          #SBATCH --account=<project-account>
          
          module load comsol
          echo "--- Copy Input Files to TMPDIR and Change Disk to TMPDIR"
          cp input_cluster.mph $TMPDIR
          cd $TMPDIR
          
          echo "--- COMSOL run"
          comsol -nn 2 batch -mpirsh ssh -inputfile input_cluster.mph -outputfile output_cluster.mph
          echo "--- Copy files back"
          cp output_cluster.mph output_cluster.mph.status ${SLURM_SUBMIT_DIR}
          echo "---Job finished at: 'date'"
          echo "---------------------------------------------"
          

          Note:

          • Set nodes to 2 and ppn to 28 ( --nodes=2 --ntasks-per-node=28). You can change the values per your need.
          • Use "-mpirsh ssh" option for multi-node jobs
          • Copy files from your directory to $TMPDIR.
          • Provide the name of the input file and output file.

          Further Reading

          Supercomputer: 
          Service: 

          Interactive Parallel COMSOL Job

          This documentation is to discuss how to set up an interactive parallel COMSOL job at OSC. The following example demonstrates the process of using COMSOL version 5.1 on Oakley. Depending on the version of COMSOL and cluster you work on, there mighe be some differences from the example. Feel free to contact OSC Help if you have any questions. 

          • Launch COMSOL GUI application following the instructions on this page. Get the information on the node(s) allocated to your job and save it in the file named hostfile using the following command:

           

          cat $PBS_NODEFILE | uniq > hostfile
          

          Make sure the hostfile is located in the same directory where you COMSOL input file is put

          • Open COMSOL GUI application. To enable the cluster compuitng feature, click the show button and select Advanced Study Options, as shown in the picture below:

          Advanced study

          • In the Model Builder, right-click a Study node and select Cluster Computing, as shown in the picture below:

          cluster computing

          • In the Cluster Computing node's setting window, select General from the Cluster type list. Provide the name of Host file as hostfile. Browse to the directory where your COMSOL input file is located, as shown in the picture below:

          setting

          • Save all the settings. Then you should be able to run an interactive parallel COMSOL job at OSC
          Supercomputer: 

          CP2K

          CP2K is a quantum chemistry and solid state physics software package that can perform atomistic simulations of solid state, liquid, molecular, periodic, material, crystal, and biological systems. CP2K provides a general framework for different modeling methods such as DFT using the mixed Gaussian and plane waves approaches GPW and GAPW. Supported theory levels include DFTB, LDA, GGA, MP2, RPA, semi-empirical methods and classical force fields. CP2K can do simulations of molecular dynamics, metadynamics, Monte Carlo, Ehrenfest dynamics, vibrational analysis, core level spectroscopy, energy minimization, and transition state optimization using NEB or dimer method.

          Availability and Restrictions

          Versions

          CP2K is available on the OSC clusters. These are the versions currently available:

          VERSION Owens Pitzer Notes
          6.1 X* X*   (owens) gnu/7.3.0 intelmpi/2018.3
            (pitzer) gnu/4.8.5 openmpi/3.1.6
            (pitzer) gnu/7.3.0 intelmpi/2018.3
            (pitzer) gnu/7.3.0 openmpi/3.1.4
          * Current default version

          You can use module spider cp2k to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          CP2K is available to all OSC users.

          Publisher/Vendor/Repository and License Type

          CP2K, GNU General Public License

          Usage

          IMPORTANT NOTE: You need to load the prerequisite compiler and MPI modules before you can load CP2K. To determine those modules, use module spider cp2k/{version}.
          We have found some types of CP2K jobs would fail or crash nodes using cp2k.popt and cp2k.psmp from MVAPICH2 and OpenMPI builds due to unstable numerical issues. It is recommended to use Intel MPI, which is more stable unless you experience any other issue.

          Usage

          Set-up

          CP2K usage is controlled via modules. Load one of the CP2K modulefiles at the command line, in your shell initialization script, or in your batch scripts. You need to load the prerequisite compiler and MPI modules before you can load CP2K. To determine those modules, use module spider cp2k/6.1

          Batch Usage

          When you log into pitzer.osc.edu you are actually logged into the login node. To gain access to the vast resources in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

          Interactive Batch Session

          For an interactive batch session one can run the following command:

          sinteractive -n 1 -t 00:20:00
          

          which requests one core (-n 1), for a walltime of 20 minutes (-t 00:20:00). You may adjust the numbers per your need.

          Non-interactive Batch Job

          batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Below is the example batch script for a parallel run:

          #!/bin/bash 
          #SLURM --nodes=2
          #SLURM --time=1:00:0
          
          module load gnu/7.3.0
          module load intelmpi/2018.4
          module load cp2k/6.1
          module list
          
          pbsdcp -p job.inp $TMPDIR
          cd $TMPDIR
          srun cp2k.popt -i job.inp -o job.out.$SLURM_JOB_ID 
          pbsdcp -g job.out.$SLURM_JOB_ID $SLURM_SUBMIT_DIR

          Further Reading

          General documentation is available from the CP2K website.

           
          Supercomputer: 
          Service: 

          CUDA

          CUDA™ (Compute Unified Device Architecture) is a parallel computing platform and programming model developed by Nvidia that enables dramatic increases in computing performance by harnessing the power of the graphics processing unit (GPU).

          Availability and Restrictions

          Versions

          CUDA is available on the clusters supporting GPUs. The versions currently available at OSC are:

          Version Owens Pitzer
          8.0.44 X  
          8.0.61 X  
          9.0.176   X
          9.1.85 X  
          9.2.88 X X
          10.0.130 X X
          10.1.168 X X
          10.2.89 X* X*
          11.0.3 X X
          * Current default version

          You can use module spider cuda to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          CUDA is available for use by all OSC users.

          Publisher/Vendor/Repository and License Type

          Nvidia, Freeware 

          Usage

          Usage on Owens

          Set-up on Owens

          To load the default version of CUDA module, use module load cuda. To select a particular software version, use   module load cuda/version

          GPU Computing SDK

          The NVIDIA GPU Computing SDK provides hundreds of code samples and covers a wide range of applications/techniques to help you get started on the path of writing software with CUDA C/C++ or DirectCompute. 

          Programming in CUDA

          Please visit the following link to learn programming in CUDA, http://developer.nvidia.com/cuda-education-training. The link also contains tutorials on optimizing CUDA codes to obtain greater speedups.

          Compiling CUDA Code

          Many of the tools loaded with the CUDA module can be used regardless of the compiler modules loaded. However, CUDA codes are compiled with nvcc, which depends on the GNU compilers. In particular, if you are trying to compile CUDA codes and encounter a compiler error such as

          #error -- unsupported GNU version! gcc versions later than X are not supported!
          

          then you need to load an older GNU compiler with the module load gnu/version command (if compiling standard C code with GNU compilers) or the module load gcc-compatibility/version command (if compiling standard C code with Intel or PGI compilers).

          One can type module show cuda-version-number to view the list of environment variables.
          To compile a cuda code contained in a file, let say mycudaApp.cu, the following could be done after loading the appropriate CUDA module: nvcc -o mycudaApp mycudaApp.cu. This will create an executable by name mycudaApp.

          The environment variable OSC_CUDA_ARCH defined in the module can be used to specify the CUDA_ARCH, to compile with nvcc -o mycudaApp -arch=$OSC_CUDA_ARCH mycudaApp.cu.

          Important: The devices are configured in exclusive mode. This means that 'cudaSetDevice' should NOT be used if requesting one GPU resource. Once the first call to CUDA is executed, the system will figure out which device it is using. If both cards per node is in use by a single application, please use 'cudaSetDevice'.

          Debugging CUDA code

          cuda-gdb can be used to debug CUDA codes. module load cuda will make it available to you. For more information on how to use the CUDA-GDB please visit http://developer.nvidia.com/cuda-gdb.

          Detecting memory access errors

          CUDA-MEMCHECK could be used for detecting the source and cause of memory access errors in your program. For more information on how to use CUDA-MEMCHECK please visit http://developer.nvidia.com/cuda-memcheck.

          Setting the GPU compute mode on Owens

          The GPUs on Owens can be set to different compute modes as listed here.

          They can be set by adding the following to the GPU specification. The exclusive-process compute mode is the default setting on our GPU nodes, so you don't need to specify. With this mode, mulitple CUDA processes across GPU nodes are not allowed, e.g CUDA processes via MPI. If you need to run a MPI-CUDA job, you need to use the default compute mode. You can specify it by using --gpu_cmode=shared, for example:

          --nodes=2 --ntasks-per-node=28 --gpus-per-node=1 --gpu_cmode=shared
          

          Batch Usage on Owens

          When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

          Interactive Batch Session

          For an interactive batch session one can run the following command:

          sinteractive -A <project-account> -N 1 -n 28 -g 1 -t 00:20:00 
          

          which requests one whole node with 28 cores (-N 1 -n 1), for a walltime of 20 minutes (-t 00:20:00), with one gpu (-g 1). You may adjust the numbers per your need.

          Non-interactive Batch Job (Serial Run)

          batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Below is the example batch script (job.txt) for a serial run:

          #!/bin/bash
          #SBATCH -- time=01:00:00
          #SBATCH --nodes=1 --ntasks-per-node=1:gpus=1
          #SBATCH --job-name compute
          #SBATCH --account=<project-account>
          
          module load cuda
          cd $HOME/cuda
          cp mycudaApp $TMPDIR
          cd $TMPDIR
          ./mycudaApp

          Usage on Pitzer

          Set-up on Pitzer

          To load the default version of CUDA module, use module load cuda.

          GPU Computing SDK

          The NVIDIA GPU Computing SDK provides hundreds of code samples and covers a wide range of applications/techniques to help you get started on the path of writing software with CUDA C/C++ or DirectCompute. 

          Programming in CUDA

          Please visit the following link to learn programming in CUDA, http://developer.nvidia.com/cuda-education-training. The link also contains tutorials on optimizing CUDA codes to obtain greater speedups.

          Compiling CUDA Code

          Many of the tools loaded with the CUDA module can be used regardless of the compiler modules loaded. However, CUDA codes are compiled with nvcc, which depends on the GNU compilers. In particular, if you are trying to compile CUDA codes and encounter a compiler error such as

          #error -- unsupported GNU version! gcc versions later than X are not supported!
          

          then you need to load an older GNU compiler with the module load gnu/version command (if compiling standard C code with GNU compilers) or the module load gcc-compatibility/version command (if compiling standard C code with Intel or PGI compilers).

          One can type module show cuda-version-number to view the list of environment variables.
          To compile a cuda code contained in a file, let say mycudaApp.cu, the following could be done after loading the appropriate CUDA module: nvcc -o mycudaApp mycudaApp.cu. This will create an executable by name mycudaApp.

          The environment variable OSC_CUDA_ARCH defined in the module can be used to specify the CUDA_ARCH, to compile with nvcc -o mycudaApp -arch=$OSC_CUDA_ARCH mycudaApp.cu.

          Important: The devices are configured in exclusive mode. This means that 'cudaSetDevice' should NOT be used if requesting one GPU resource. Once the first call to CUDA is executed, the system will figure out which device it is using. If both cards per node is in use by a single application, please use 'cudaSetDevice'.

          Debugging CUDA code

          cuda-gdb can be used to debug CUDA codes. module load cuda will make it available to you. For more information on how to use the CUDA-GDB please visit http://developer.nvidia.com/cuda-gdb.

          Detecting memory access errors

          CUDA-MEMCHECK could be used for detecting the source and cause of memory access errors in your program. For more information on how to use CUDA-MEMCHECK please visit http://developer.nvidia.com/cuda-memcheck.

          Setting the GPU compute mode on Pitzer

          The GPUs on Pitzer can be set to different compute modes as listed here.

          The exclusive-process compute mode is the default setting on our GPU nodes, so you don't need to specify. With this mode, mulitple CUDA processes across GPU nodes are not allowed, e.g CUDA processes via MPI. If you need to run a MPI-CUDA job, you need to use the default compute mode. You can specify it by using --gpu_cmode=shared, for example:

          --nodes=1 --ntasks-per-node=40 --gpus-per-node=2 --gpu_cmode=shared
          

          Batch Usage on Pitzer

          When you log into pitzer.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

          Interactive Batch Session

          For an interactive batch session one can run the following command:

          sinteractive -A <project-account> -N 1 -n 40 -g 2 -t 00:20:00
          

          which requests one whole node (-N 1), 40 cores (-n 40), 2 gpus (-g 2), and a walltime of 20 minutes (-t 00:20:00). You may adjust the numbers per your need.

          Non-interactive Batch Job (Serial Run)

          batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Below is the example batch script (job.txt) for a serial run:

          #!/bin/bash
          #SBATCH --time=01:00:00
          #SBATCH --nodes=1 --ntasks-per-node=1 --gpus-per-node=1
          #SBATCH --job-name Compute
          #SBATCH --account=<project-account>
          
          module load cuda
          cd $HOME/cuda
          cp mycudaApp $TMPDIR
          cd $TMPDIR
          ./mycudaApp
          

          GNU Compiler Support for NVCC 

          CUDA Version GNU Version
          9.2.88 - 10.0.130 Up to GCC 7
          10.1.168 - 10.2.89 Up to GCC 8
          11.0.3 -  Up to GCC 9

          Further Reading

          Online documentation is available on the CUDA homepage.

          Compiler support for the latest version of CUDA is available here.

          CUDA optimization techniques.

           

          Supercomputer: 
          Service: 
          Technologies: 
          Fields of Science: 

          Caffe

          Caffe is "a fast open framework for deep learning."

          From their README:

          Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by the Berkeley Vision and Learning Center (BVLC) and by community contributors. Yangqing Jia created  the project during his PhD at UC Berkeley. Caffe is released under the BSD 2-Clause license.
          

          Caffe also includes interfaces for both Python and Matlab, which have been built but have not been tested.

          Availability and Restrictions

          Versions

          The following versions of Caffe are available on OSC clusters:

          Version Owens
          1.0.0-rc3 X*
          * Current Default Version

          You can use module spider caffe to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          The current version of Caffe on Owens requires cuda/8.0.44 for GPU calculations.

          Access 

          Caffe is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Berkeley AI Research, Open source

          Usage

          Usage on Owens

          Setup on Owens

          To configure the Owens cluster for the use of Caffe, use the following commands:

          module load caffe
          

          Batch Usage on Owens

          Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations for Owens, and Scheduling Policies and Limits for more info.  In particular, Caffe should be run on a GPU-enabled compute node.

          An Example of Using Caffe with MNIST Training Data on Owens

          Below is an example batch script (job.txt) for using Caffe, see this resource for a detailed explanation http://caffe.berkeleyvision.org/gathered/examples/mnist.html

          #!/bin/bash
          #PBS -N Caffe
          #PBS -l nodes=1:ppn=28:gpu
          #PBS -l walltime=30:00
          #PBS -j oe
          #PBS -S /bin/bash 
          cd $PBS_O_WORKDIR
          . /etc/profile.d/lmod.sh
          # Load the modules for Caffe
          ml caffe
          # Migrate to job temp directory and copy folders over
          cd $TMPDIR
          cp -r $CAFFE_HOME/{examples,data} .
          # Download, create, train
          ./data/mnist/get_mnist.sh
          ./examples/mnist/create_mnist.sh
          ./examples/mnist/train_lenet.sh
          # Serialize log files
          echo; echo 'LOG 1'
          cat convert_mnist_data.bin.$(hostname)*
          echo; echo 'LOG 2'
          cat caffe.INFO
          echo; echo 'LOG 3'
          cat convert_mnist_data.bin.INFO
          echo; echo 'LOG 4'
          cat caffe.$(hostname).*
          cp examples/mnist/lenet_iter_10000.* $PBS_O_WORKDIR
          

          In order to run it via the batch system, submit the job.txt file with the following command:

          qsub job.txt
          

          Further Reading

          Supercomputer: 
          Service: 
          Technologies: 
          Fields of Science: 

          Clustal W

          Clustal W is a multiple sequence alignment program written in C++.

          Availability and Restrictions

          Versions

          The following versions of bedtools are available on OSC clusters:

          Version Owens Pitzer
          2.1 X* X*
          * Current default version

          You can use module spider clustalw to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          Clustal W is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          GNU Lesser GPL.

          Usage

          Usage on Owens

          Set-up

          To configure your environment for use of metilene, run the following command: module load clustalw. The default version will be loaded. To select a particular MUSCLE version, use module load clustalw/version. For example, use module load clustalw/2.1 to load Clustal W 2.1.

          Usage on Pitzer

          Set-up

          To configure your environment for use of metilene, run the following command: module load clustalw. The default version will be loaded. To select a particular MUSCLE version, use module load clustalw/version. For example, use module load clustalw/2.1 to load Clustal W 2.1.

          Further Reading

          Supercomputer: 
          Service: 
          Fields of Science: 

          Connectome Workbench

          Connectome Workbench is an open source, freely available visualization and discovery tool used to map neuroimaging data, especially data generated by the Human Connectome Project.

          Availability and Restrictions

          Versions

          Connectome Workbench is available on Owens and Pitzer clusters. These are the versions currently available:

          Version Owens Pitzer Notes
          1.3.2  X* X*  
          * Current default version

          You can use module spider connectome-workbench to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          Connectome Workbench is available to all OSC users.

          Publisher/Vendor/Repository and License Type

          Washington University School of Medicine, GPL

          Usage

          Set-up

          To configure your environment for use of Bowtie1, run the following command:  module load connectome-workbench. The default version will be loaded. To select a particular Bowtie1 version, use  module load connectome-workbench/version. For example, use  module load connectome-workbench/1.3.2 to load Connectome Workbench 1.3.2.

          Further Reading

          General documentation is available from the Connectome Workbench hompage.

           
          Supercomputer: 
          Service: 
          Fields of Science: 

          Cufflinks

          Cufflinks is a program that analyzes RNA -Seq samples. It assembles aligned RNA-Seq reads into a set of transcripts, then inspects the transcripts to estimate abundances and test for differential expression and regulation in the RNA-Seq reads.

          Availability and Restrictions

          Versions

          Cufflinks is available on the Owens Cluster. The versions currently available at OSC are:

          Version Owens
          2.2.1 X*
          * Current Default Version

          You can use module spider cufflinks to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          Cufflinks is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Cole Trapnell et al., Open source

          Usage

          Usage on Owens

          Set-up

          To configure your enviorment for use of Cufflinks, use command module load cufflinks. This will load the default version.

          Further Reading

          Supercomputer: 
          Fields of Science: 

          Darshan

          Darshan is a lightweight "scalable HPC I/O characterization tool".  It is intended to profile I/O by emitting log files to a consistent log location for systems administrators, and also provides scripts to create summary PDFs to characterize I/O in MPI-based programs.

          Availability and Restrictions

          Versions

          The following versions of Darshan are available on OSC clusters:

          Version Owens Pitzer
          3.1.2 X  
          3.1.4 X  
          3.1.5-pre1 X  
          3.1.5 X  
          3.1.6 X X
          3.1.8 X* X*
          3.2.1 X X
          * Current default version

          You can use module spider darshan to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access 

          Darshan is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          MCSD, Argonne National Laboratory, Open source

          Usage

          Usage on Owens

          Setup on Owens

          To configure the Owens cluster for Darshan use the following commands:

          module load darshan

          Darshan is only supported for the following compiler and MPI implementations:

          gnu/4.8.5  mvapich2/2.2
          gnu/4.8.5  mvapich2/2.2rc1
          gnu/4.8.5  openmpi/1.10
          intel/16.0.3  intelmpi/5.1.3
          intel/16.0.3  mvapich2/2.2
          intel/16.0.3  mvapich2/2.2rc1
          intel/16.0.3  openmpi/1.10
          

          Batch Usage on Owens

          Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations for Owens, and Scheduling Policies and Limits for more info. 

          If you have an MPI-based program the syntax is as simple as

          # basic call to darshan 
          mpiexec.darshan [args] ./my_mpi_program
          # to show evidence that Darshan is working and to see internal timing
          mpiexec.darshan.timing [args] ./my_mpi_program
          
          An Example of Using Darshan with MPI-IO

          Below is an example batch script (mpiio_with_darshan.qsub) for understanding MPI-IO, see this resource for a detailed explanation: http://beige.ucs.indiana.edu/I590/node29.html.  The C program examples have each MPI task write to the same file at different offset locations sequentially.  A serial version (1 processor writes to 1 file) is included and timed for comparison.  Because the files generated here are large scratch files there is no need to retain them.

          1. Load the Darshan module
          2. Create a function as a build and run harness for each case
          3. Run the cases
          4. Generate PDF reports (using pdflatex and supporting files) from binary log files.
          5. Check job output and read PDF reports.
          #!/bin/bash
          #PBS -l nodes=1:ppn=28:pfsdir
          #PBS -j oe
          # one may need to perform 'shopt -s expand_aliases' inside the shell before calling this script to expand darshan aliases
          shopt -s expand_aliases
          ml r --quiet
          module load darshan
          ml
          function run_darshan() {
          COMPILER=mpicc
          MPIPROCS=$1
          PROGNAME=$2
          BLOKSIZE=$3
          LOCATION=$4
          LOGNAME=$5
          export DARSHAN_LOGFILE=$TMPDIR/$LOGNAME
          cp ${DARSHAN_EXAMPLE}/${PROGNAME}.c $LOCATION
          cd $LOCATION
          $COMPILER -DDEBUG -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -o $PROGNAME ${PROGNAME}.c
          mpiexec_darshan() { eval mpiexec.darshan "$@"; }
          mpiexec_darshan -n $MPIPROCS ./$PROGNAME -f test -l $BLOKSIZE
          du -sh test # show the human-readable file size
          rm test # we are not keeping the large file around
          }
          JOBID=$([[ $tmp =~ ([0-9]*)\..* ]]; echo "${BASH_REMATCH[1]}") 
          # run the parallel version of mpi-io to write to a file but use the local tmp directory
          run_darshan $PBS_NP mkrandpfile 256 $TMPDIR ${JOBID}_tmpdir.darshan
          # run the parallel version of mpi-io to write to a file but use the parallel file system scratch location
          run_darshan $PBS_NP mkrandpfile 256 $PFSDIR ${JOBID}_pfsdir.darshan
          # for each darshan log generate a PDF report
          cd $TMPDIR
          for log in $(ls *.darshan); do
            darshan-job-summary.pl $log
          done
          cp *.pdf $PBS_O_WORKDIR
          

          In order to run it via the batch system, submit the mpiio_with_darshan.qsub file with the following command:

          qsub mpiio_with_darshan.qsub
          

          Usage on Pitzer

          Setup on Pitzer

          To configure the Pitzer cluster for Darshan use the following commands:

          module load darshan

          Darshan is only supported for the following compiler and MPI implementations:

          intel/18.0.3  mvapich2/2.3
          intel/18.0.4  mvapich2/2.3
          

          Batch Usage on Pitzer

          Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems.

          If you have an MPI-based program the syntax is as simple as

          # basic call to darshan 
          mpiexec.darshan [args] ./my_mpi_program
          # to show evidence that Darshan is working and to see internal timing
          mpiexec.darshan.timing [args] ./my_mpi_program
          
          An Example of Using Darshan with MPI-IO

          Below is an example batch script (mpiio_with_darshan.qsub).  The C program examples have each MPI task write to the same file at different offset locations sequentially.  A serial version (1 processor writes to 1 file) is included and timed for comparison.  Because the files generated here are large scratch files there is no need to retain them.

          1. Load the Darshan module
          2. Create a function as a build and run harness for each case
          3. Run the cases
          4. Generate PDF reports (using pdflatex and supporting files) from binary log files.
          5. Check job output and read PDF reports.
          #!/bin/bash
          #PBS -l nodes=1:ppn=40:pfsdir
          #PBS -j oe
          # one may need to perform 'shopt -s expand_aliases' inside the shell before calling this script to expand darshan aliases
          shopt -s expand_aliases
          ml r --quiet
          module load darshan
          ml
          function run_darshan() {
          COMPILER=mpicc
          MPIPROCS=$1
          PROGNAME=$2
          BLOKSIZE=$3
          LOCATION=$4
          LOGNAME=$5
          export DARSHAN_LOGFILE=$TMPDIR/$LOGNAME
          cp ${DARSHAN_EXAMPLE}/${PROGNAME}.c $LOCATION
          cd $LOCATION
          $COMPILER -DDEBUG -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -o $PROGNAME ${PROGNAME}.c
          mpiexec_darshan() { eval mpiexec.darshan "$@"; }
          mpiexec_darshan -n $MPIPROCS ./$PROGNAME -f test -l $BLOKSIZE
          du -sh test # show the human-readable file size
          rm test # we are not keeping the large file around
          }
          JOBID=$([[ $tmp =~ ([0-9]*)\..* ]]; echo "${BASH_REMATCH[1]}") 
          # run the parallel version of mpi-io to write to a file but use the local tmp directory
          run_darshan $PBS_NP mkrandpfile 256 $TMPDIR ${JOBID}_tmpdir.darshan 
          # run the parallel version of mpi-io to write to a file but use the parallel file system scratch location
          run_darshan $PBS_NP mkrandpfile 256 $PFSDIR ${JOBID}_pfsdir.darshan 
          # for each darshan log generate a PDF report
          cd $TMPDIR
          for log in $(ls *.darshan); do
            darshan-job-summary.pl $log
          done
          cp *.pdf $PBS_O_WORKDIR
          

          In order to run it via the batch system, submit the mpiio_with_darshan.qsub file with the following command:

          qsub mpiio_with_darshan.qsub

          Further Reading

          Supercomputer: 
          Service: 
          Technologies: 
          Fields of Science: 

          Desmond

          Desmond is a software package that perform high-speed molecular dynamics simulations of biological systems on conventional commodity clusters, general-purpose supercomputers, and GPUs. The code uses novel parallel algorithms and numerical techniques to achieve high performance and accuracy on platforms containing a large number of processors, but may also be executed on a single computer. Desmond includes code optimized for machines with an NVIDIA GPU.

          Availability and Restrictions

          Versions

          The Desmond package is available on Owens. The versions currently available at OSC are:

          Version Owens
          2018.2 X
          2019.1 X*
          * Current default version

          You can use  module spider desmond to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access for Academic Users 

          Desmond is available to academic OSC users. Please review the license agreement carefully before use. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          D E Shaw Research, Non-Commercial

          Usage

          Usage on Owens

          To set up your environment for desmond load one of its module files:
           
          ​​module load desmond/2018.2

           

          Desmond comes with Schrodinger interactive builder, Maestro.  To run maestro, connect to OSC OnDemand and luanch a virtual desktop, either a Virtual Desktop Interface (VDI) or an Interactive HPC Desktop, and in desktop open a terminal and run:

          maestro 
          
          Here is an example batch script that uses Desmond non-interactively via the batch system:

           

          #!/bin/bash
          #SBATCH --job-name multisim-batch
          #SBATCH --time=0:20:00
          #SBATCH --nodes=1 --ntasks-per-node
          #SBATCH --acount=<project-account>
          
          # Example Desmond single-node batch script. 
          
          sstat -j $SLURM_JOB_ID
          export
          module load desmond
          module list
          
          cp --preserve desmond_md_job_butane.* $TMPDIR
          
          cd $TMPDIR
          $SCHRODINGER/utilities/multisim -HOST localhost -maxjob 1 -cpu 24 -m desmond_md_job_butane.msj -c desmond_md_job_butane.cfg desmond_md_job_butane.cms -mode umbrella -ATTACHED -WAIT
          
          ls -l
          pbsdcp --preserve --recursive --gather '*' $SLURM_SUBMIT_DIR

          The WAIT option forces the multisim command to wait until all tasks of the command are completed. This is necessary for PBS batch jobs to run effectively. The HOST option specifies how tasks are distributed over processors.

          Further Reading

          Supercomputer: 
          Service: 

          FASTX-Toolkit

          The FASTX-Toolkit is a collection of command line tools for Short-Reads FASTA/FASTQ files preprocessing.

          Availability and Restrictions

          Verisons

          The following versions of FASTX-Toolkit are available on OSC clusters:

          Version Owens
          0.0.14 X*
          *:Current default version

          You can use  module spider fastx to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          FASTX-Toolkit is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Assaf Gordon, Open source

          Usage

          Usage on Owens

          Set-up

          To configure your environment for use of FASTX-Toolkit, run the following command: module load fastx. The default version will be loaded. To select a particular FASTX-Toolkit version, use module load fastx/version. For example, use module load fastx/0.0.14 to load FASTX-Toolkit 0.0.14.

          Further Reading

          Supercomputer: 
          Service: 
          Fields of Science: 

          FFTW

          FFTW is a C subroutine library for computing the Discrete Fourier Transform (DFT) in one or more dimensions, of arbitrary input size, and of both real and complex data. It is portable and performs well on a wide variety of platforms.

          Availability and Restrictions

          Versions

          FFTW is available on Ruby, and Owens Clusters. The versions currently available at OSC are:

          Version Owens Pitzer
          3.3.4 X  
          3.3.5 X  
          3.3.8  X* X*
          NOTE: FFTW2 and FFTW3 are tracked separately in the module system

          You can use module spider fftw2  or module spider fftw3  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          FFTW is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          www.fftw.org, Open source

          Usage

          Usage on Owens

          Set-up

          Initalizing the system for use of the FFTW library is dependent on the system you are using and the compiler you are using. A successful build of your program will depend on an understanding of what module fits your circumstances. To load a particular version, use  module load name . For example, use  module load fftw3/3.3.4  to load FFTW3 version 3.3.4. You can use module spider fftw  to view available modules.

          Building with FFTW

          The following environment variables are setup when the FFTW library is loaded:

          Variable Use
          $FFTW3_CFLAGS Use during your compilation step for C programs.
          $FFTW3_FFLAGS Use during your compilation step for Fortran programs.
          $FFTW3_LIBS Use during your link step for the sequential version of the library.
          $FFTW3_LIBS_OMP Use during your link step for the OpenMP version of the library.
          $FFTW3_LIBS_MPI Use during your link step for the MPI version of the library.
          $FFTW3_LIBS_THREADS Use during your link step for the "threads" version of the library.

          below is a set of example commands used to build a file called my-fftw.c .

          module load fftw3
          icc $FFTW3_CFLAGS my-fftw.c -o my-fftw $FFTW3_LIBS 
          ifort $FFTW3_FFLAGS more-fftw.f -o more-fftw $FFTW3_LIBS
          

          Usage on Pitzer

          Set-up

          Initalizing the system for use of the FFTW library is dependent on the system you are using and the compiler you are using. A successful build of your program will depend on an understanding of what module fits your circumstances. To load a particular version, use  module load fftw3/.

          Building with FFTW

          The following environment variables are setup when the FFTW library is loaded:

          VARIABLE USE
          $FFTW3_CFLAGS Use during your compilation step for C programs.
          $FFTW3_FFLAGS Use during your compilation step for Fortran programs.
          $FFTW3_LIBS Use during your link step for the sequential version of the library.
          $FFTW3_LIBS_OMP Use during your link step for the OpenMP version of the library.
          $FFTW3_LIBS_MPI Use during your link step for the MPI version of the library.

          below is a set of example commands used to build a file called my-fftw.c .

          module load fftw3
          icc $FFTW3_CFLAGS my-fftw.c -o my-fftw $FFTW3_LIBS 
          ifort $FFTW3_FFLAGS more-fftw.f -o more-fftw $FFTW3_LIBS

          Further Reading

          See Also

          Supercomputer: 
          Service: 

          FSL

          FSL is a library of tools for analyzing FMRI, MRI and DTI brain imaging data.

          Availability and Restrictions

          Versions

          The following versions of FSL are available on OSC clusters:

          Version Owens Pitzer
          5.0.10

          X*

           
          6.0.4 X X
          * Curent default version

          You can use module spider fsl to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          FSL is available to academic OSC users. Please review the license agreement carefully before use. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Analysis Group, University of Oxford/ freeware

          Usage

          Usage on Owens and Pitzer

          Set-up

          Configure your environment for use of FSL with module load fsl. This will load the default version.

          Using FSL GUI

          Access the FSL GUI with command for bash

          source $FSLDIR/etc/fslconf/fsl.sh
          fsl

          For csh, one can use

          source $FSLDIR/etc/fslconf/fsl.csh
          fsl 

          This will bring up a menu of all FSL tools. For information on individual FSL tools see FSL Overview page.

          Further Reading 

          Supercomputer: 
          Fields of Science: 

          FastQC

          FastQC provides quality control checks of high throughput sequence data that identify areas of the data that may cause problems during further analysis.

          Availability and Restrictions

          Versions

          FastQC is available on the Owens cluster. The versions currently available at OSC are:

          Version Owens Pitzer
          0.11.5 X*  
          0.11.7 X  
          0.11.8   X*
          * Current Default Version

          You can use module spider fastqc to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          FastQC is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Babraham Bioinformatics, Open source

          Usage

          Usage on Owens

          Set-up

          To configure your enviorment for use of FastQC, use command module load fastqc. This will load the default version.

          Usage on Pitzer

          Set-up

          To configure your enviorment for use of FastQC, use command module load fastqc. This will load the default version.

          Further Reading

          Supercomputer: 
          Fields of Science: 

          FreeSurfer

          FreeSurfer is a software package used to anaylze nueroimaging data.

          Availability & Restrictions

          Versions

          The following versions of FreeSurfer are available on OSC clusters:

          Version Owens Pitzer Note
          5.3.0 X    
          6.0.0

           X*

             
          7.1.1 X X*  
          * Curent default version

          You can use module spider freesurfer to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          FreeSurfer is available to academic OSC users. Please review the license agreement carefully before use. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Athinoula A. Martinos Center, Open source

          Usage

          Usage on Owens and Pitzer

          Set-up

          Load the FreeSurfer module with  module load freesurfer. This will load the default version. Then, to continue configuring your environment, you must source the setup script for Freesurfer. Do this with the following command that corresponds to the Linux shell you are using. If using bash, use:

          source $FREESURFER_HOME/SetUpFreeSurfer.sh
          

          If using tcsh, use:

          source $FREESURFER_HOME/SetUpFreeSurfer.csh
          

          To finish configuring FreeSurfer, set the the FreeSurfer environment variable SUBJECTS_DIR to the directory of your subject data. The SUBJECTS_DIR variable defaults to the FREESURFER_HOME/subjects directory, so if this is your intended directory to use the enviornment set-up is complete.

          To alter the SUBJECTS_DIR variable, however, again use the following command that corresponds to the Linux shell you are using. For bash:

          export SUBJECTS_DIR=<path to subject data>
          

          For tcsh:

          setenv SUBJECTS_DIR=<path to subject data>
          

          Note that you can set the SUBJECT_DIR variable before or after sourcing the setup script.

          The cuda applications from FreeSurfer requires CUDA 5 library (which is not avaiable through module system). To set up cuda environment, run the following command after load the FreeSurfer module. If  you are using bash, run:

          source $FREESURFER_HOME/bin/cuda5_setup.sh
          

          If using tcsh, use:

          source $FREESURFER_HOME/bin/cuda5_setup.csh
          

          Further Reading 

          Supercomputer: 
          Service: 
          Fields of Science: 

          GAMESS

          The General Atomic and Molecular Electronic Structure System (GAMESS) is a flexible ab initio electronic structure program. Its latest version can perform general valence bond, multiconfiguration self-consistent field, Möller-Plesset, coupled-cluster, and configuration interaction calculations. Geometry optimizations, vibrational frequencies, thermodynamic properties, and solution modeling are available. It performs well on open shell and excited state systems and can model relativistic effects. The GAMESS Home Page has additional information.

          Availability and Restrictions

          Versions

          The current versions of GAMESS available on the Oakley and Owens Clusters are:

          VERSION

          owens Pitzer
          18 AUG 2016 (R1) X  
          30 Sep 2019 (R2) X* X*
          * Current default version

          You can use module spider gamess to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          GAMESS is available to all OSC users. Please review the license agreement carefully before use. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Gordon research group, Iowa State Univ./ Proprietary freeware

          Usage

          Set-up

          GAMESS usage is controlled via modules. Load one of the GAMESS modulefiles at the command line, in your shell initialization script, or in your batch scripts, for example:

          module load gamess  

          Examples

          Further Reading

          General documentation is available from the GAMESS Home page and in the local machine directories.

          Supercomputer: 
          Service: 

          GATK

          GATK is a software package for analysis of high-throughput sequencing data. The toolkit offers a wide variety of tools, with a primary focus on variant discovery and genotyping as well as strong emphasis on data quality assurance.

          Availability and Restrictions

          Versions

          The following versions of GATK are available on OSC clusters:

          Version Owens Pitzer Notes
          3.5 X    
          4.0.11.0   X  
          4.1.2.0 X* X*  
          * Current default version

          You can use module spider gatk to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access for Academic Users

          GATK4 is available to all OSC users under BSD 3-clause License.

          GATK3 is available to academic OSC users. Please review the license agreement carefully before use. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Broad Institute, Inc., BSD 3-clause License (GATK4 only)

          Usage

          Usage on Owens

          Set-up

          To configure your environment for use of GATK, run the following command: module load gatk. The default version will be loaded. To select a particular GATK version, use module load gatk/version. For example, use module load gatk/4.1.2.0 to load GATK 4.1.2.0.

          Usage

          This software is a Java executable .jar file; thus, it is not possible to add to the PATH environment variable. From module load gatk, a new environment variable, GATK, will be set. Thus, users can use the software by running the following command: gatk {other options},e.g  run gatk -h to see all options.

          Usage on Pitzer

          Set-up

          To configure your environment for use of GATK, run the following command: module load gatk. The default version will be loaded.

          Usage

          This software is a Java executable .jar file; thus, it is not possible to add to the PATH environment variable. From module load gatk, a new environment variable, GATK, will be set. Thus, users can use the software by running the following command: gatk {other options},e.g  run gatk -h to see all options.

          Known Issues

          CBLAS undefined symbol error

          Update: 05/22/2019 
          Version: all

          If you use GATK tools that need CBLAS (e.g. CreateReadCountPanelOfNormals), you might encounter an error as

          INFO: successfully loaded /tmp/jniloader1239007313705592313netlib-native_system-linux-x86_64.so
          java: symbol lookup error: /tmp/jniloader1239007313705592313netlib-native_system-linux-x86_64.so: undefined symbol: cblas_dspr
          java: symbol lookup error: /tmp/jniloader1239007313705592313netlib-native_system-linux-x86_64.so: undefined symbol: cblas_dspr

          The error raises because the system-default LAPACK does not support CBLAS.  The remedy is to run GATK in conjunction with lapack/3.8.0:

          $ module load lapack/3.8.0
          $ module load gatk/4.1.2.0
          $ LD_LIBRARY_PATH=$OSC_LAPACK_DIR/lib64 gatk AnyTool toolArgs
          

          Alternatively, we recommend using the GATK container. First, download the GATK container to your home or project directory

          $ qsub -I -l nodes=1:ppn=1
          $ cd $TMPDIR
          $ export SINGULARITY_CACHEDIR=$TMPDIR
          $ SINGULARITY_TMPDIR=$TMPDIR 
          $ singularity pull docker://broadinstitute/gatk:4.1.2.0
          $ cp gatk_4.1.2.0.sif ~/

          Then run any GATK tool via

          $ singularity exec ~/gatk_4.1.2.0.sif gatk AnyTool ToolArgs
          

          You can read more about container in general from here. If you have any further questions, please contact OSC Help.

          Further Reading

          Supercomputer: 
          Service: 
          Fields of Science: 

          GLPK

          GLPK (GNU Linear Programming Kit) is a set of open source LP (linear programming) and MIP (mixed integer problem) routines written in ANSI C, which can be called from within C programs. 

          Availability and Restrictions

          Versions

          The following versions are available on OSC systems:

          Version Owens
          4.60 X*
          * Current default version

          You can use module spider glpk to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          GLPK is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          GNU, Open source

          Usage

          Set-up

          To set up your environment for using GLPK on Oakley, run the following command:

          module load glpk

          Compiling and Linking

          To compile your C code using GLPK API routines, use the environment variable $GLPK_CFLAGS provided by the module:

          gcc $GLPK_CFLAGS -c my_prog.c

          To link your code, use the variable $GLPK_LIBS:

          gcc my_prog.o $GLPK_LIBS -o my_prog

          glpsol

          Additionally, the GLPK module contains a stand-alone LP/MIP solver, which can be used to process files written in the GNU MathProg modeling language.  The solver can be invoked using the following command syntax:

          glpsol [options] [filename]

          For a complete list of options, use the following command:

          glpsol --help

          Further Reading

          Supercomputer: 
          Service: 

          GMAP

          GMAP is a genomic mapping and alignment program for mRNA and EST sequences.

          Availability and Restrictions

          Versions

          The following versions of GMAP are available on OSC clusters:

          Version Owens
          2016-06-09 X*
          * Current default version

          You can use module spider gmap to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          GMAP is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Genentech, Inc., Open source

          Usage

          Usage on Owens

          Set-up

          To configure your environment for use of GMAP, run the following command: module load gmap. The default version will be loaded. To select a particular GMAP version, use module load gmap/version. For example, use module load gmap/2016-06-09 to load GMAP 2016-06-09.

          Further Reading

          Supercomputer: 
          Service: 
          Fields of Science: 

          GNU Compilers

          Fortran, C and C++ compilers produced by the GNU Project. 

          Availability and Restrictions

          Versions

          GNU compilers are available on all our clusters. These are the versions currently available:

          Version Owens Pitzer notes
          4.8.5 X# X **See note below.
          4.9.1      
          5.2.0      
          6.1.0 X    
          6.3.0 X   Libraries have been built for this version
          7.3.0 X X  
          8.1.0   X  
          9.1.0 X* X*  
          * Current Default Version; # System version
          ** There is always some version of the GNU compilers in the environment. If you want a specific version you should load the appropriate module. If you don't have a gnu module loaded you will get either the system version or some other version, depending on what modules you do have loaded.

          You can use module spider gnu to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          To find out what version of gcc you are using, type gcc --version.

          Access

          The GNU compilers are available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          https://www.gnu.org/software/gcc/, Open source

          Usage

          Usage on Owens

          Set-up

          To configure your environment for use of the GNU compilers, run the following command (you may have to unload your selected compiler - if an error message appears, it will provide instructions): module load gnu.  The default version will be loaded. To select a particular GNU version, use module load gnu/version. For example, use module load gnu/4.8.5 to load GNU 4.8.5.

          How to Compile

          Once the module is loaded, follow the guides below for compile commands:

          Language non-mpi mpi
          Fortran 90 or 95 gfortran mpif90
          Fortran 77 gfortran mpif77
          c gcc mpicc
          c++ g++ mpicxx

          Building Options

          The GNU compilers recognize the following command line options :

          Compiler Option Purpose
          -fopenmp Enables compiler recognition of OpenMP directives (except mpif77)
          -o FILENAME

          Specifies the name of the object file

          -O0 or no -O  option Disable optimization
          -O1 or -O Ligh optimization
          -O2 Heavy optimization
          -O3 Most expensive optimization (Recommended)

          There are numerous flags that can be used. For more information run man <compiler binary name>.

          Batch Usage

          When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

          Interactive Batch Session
          For an interactive batch session, one can run the following command:
          sinteractive -A <project-account> -N 1 -n 28 -l -t 1:00:00
          
          which gives you 1 node and 28 cores (-N 1 -n 28),  with 1 hour (-t 1:00:00).  You may adjust the numbers per your need.
          Non-interactive Batch Job (Serial Run)
          batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will use the input file named hello.c and the output file named hello_results
          Below is the example batch script (job.txt) for a serial run:
          #!/bin/bash
          #SBATCH --time=1:00:00
          #SBATCH --nodes=1 --ntasks-per-node=28
          #SBATCH --job-name jobname
          #SBATCH --account=<project-account>
          
          module load gnu
          cp hello.c $TMPDIR
          cd $TMPDIR
          gcc -O3 hello.c -o hello
          ./hello > hello_results
          cp hello_results $SLURM_SUBMIT_DIR
          

          In order to run it via the batch system, submit the job.txt file with the following command:

          sbatch job.txt
          
          Non-interactive Batch Job (Parallel Run)
          Below is the example batch script (job.txt) for a parallel run:
          #!/bin/bash
          #SBATCH --time=1:00:00
          #SBATCH --nodes=2 --ntasks-per-node=28 
          #SBATCH --job-name jobname
          #SBATCH --account=<project-account>
          
          module load gnu
          mpicc -O3 hello.c -o hello
          cp hello $TMPDIR
          cd $TMPDIR
          mpiexec ./hello > hello_results
          cp hello_results $SLURM_SUBMIT_DIR
          

          Usage on Pitzer

          Set-up

          To configure your environment for use of the GNU compilers, run the following command (you may have to unload your selected compiler - if an error message appears, it will provide instructions): module load gnu.  The default version will be loaded. To select a particular GNU version, use module load gnu/version. For example, use module load gnu/8.1.0 to load GNU 8.1.0.

          How to Compile

          Once the module is loaded, follow the guides below for compile commands:

          LANGUAGE NON-MPI MPI
          Fortran 90 or 95 gfortran mpif90
          Fortran 77 gfortran mpif77
          c gcc mpicc
          c++ g++ mpicxx

          Building Options

          The GNU compilers recognize the following command line options :

          COMPILER OPTION PURPOSE
          -fopenmp Enables compiler recognition of OpenMP directives (except mpif77)
          -o FILENAME

          Specifies the name of the object file

          -O0 or no -O  option Disable optimization
          -O1 or -O Ligh optimization
          -O2 Heavy optimization
          -O3 Most expensive optimization (Recommended)

          There are numerous flags that can be used. For more information run man <compiler binary name>.

          Further Reading

          See Also

          Supercomputer: 
          Service: 
          Technologies: 
          Fields of Science: 

          GROMACS

          GROMACS is a versatile package of molecular dynamics simulation programs. It is primarily designed for biochemical molecules, but it has also been used on non-biological systems.  GROMACS generally scales well on OSC platforms. Starting with version 4.6 GROMACS includes GPU acceleration.

          Availability and Restrictions

          Versions

          GROMACS is available on Pitzer and Owens Clusters. Both single and double precision executables are installed. The versions currently available at OSC are the following:

          Version Owens Pitzer Notes
          5.1.2 SPC   Default version on Owens prior to 09/04/2018
          2016.4 SPC    
          2018.2 SPC SPC  
          2020.2 SPC* SPC*  
          * Current default version; S = serial single node executables; P = parallel multinode; C = CUDA (GPU)

          You can use module spider gromacs  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          GROMACS is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          http://www.gromacs.org/ Open source

          Usage 

          Usage on Owens

          Set-up

          To load the module for the default version of GROMACS, which initializes your environment for the GROMACS application, use module load gromacs. To select a particular software version, use module load gromacs/version. For example, use module load gromacs/5.1.2 to load GROMACS version 5.1.2; and use module help gromacs/5.1.2 to view details, such as, compiler prerequisites, additional modules required for specific executables, the suffixes of executables, etc.; some versions require specific prerequisite modules, and such details may be obtained with the command   module spider gromacs/version.

          Using GROMACS

          To execute a serial GROMACS versions 5 program interactively, simply run it on the command line, e.g.:

          gmx pdb2gmx
          

          Parallel multinode GROMACS versions 5 programs should be run in a batch environment with srun, e.g.:

          srun gmx_mpi_d mdrun
          

          Note that '_mpi' indicates a parallel executable and '_d' indicates a program built with double precision ('_gpu' denotes a GPU executable built with CUDA).  See the module help output for specific versions for more details on executable naming conventions.

          Batch Usage

          When you log into Owens you are actually connected to a login node. To  access the compute nodes, you must submit a job to the batch system for execution. Batch jobs can request multiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

          Interactive Batch Session
          For an interactive batch session on Owens, one can run the following command:
          sinteractive -A <project-account> -N 1 -n 28 -t 1:00:00
          
          which gives you one node with 28 cores (-N 1 -n 28), with 1 hour (-t 1:00:00). You may adjust the numbers per your need.
          Non-interactive Batch Job (Parallel Run)

          batch script can be created and submitted for a serial, cuda (GPU), or parallel run. You can create the batch script using any text editor in a working directory on the system of your choice. Sample batch scripts and input files for all types of hardware resources are available here:

          ~srb/workshops/compchem/gromacs/
          

          This simple batch script demonstrates some important points:

          #!/bin/bash
          # GROMACS Tutorial for Solvation Study of Spider Toxin Peptide
          # see fwspider_tutor.pdf
          #SBATCH --job-name fwsinvacuo.owens
          #SBATCH --nodes=2 --ntasks-per-node=28
          #SBATCH --account=<project-account>
          
          module load gromacs
          pbsdcp -p 1OMB.pdb em.mdp $TMPDIR
          # Use TMPDIR for best performance.
          cd $TMPDIR
          pdb2gmx -ignh -ff gromos43a1 -f 1OMB.pdb -o fws.gro -p fws.top -water none
          editconf -f fws.gro -d 0.7
          editconf -f out.gro -o fws_ctr.gro -center 2.0715 1.6745 1.914
          grompp -f em.mdp -c fws_ctr.gro -p fws.top -o fws_em.tpr
          srun gmx_mpi mdrun -s fws_em.tpr -o fws_em.trr -c fws_ctr.gro -g em.log -e em.edr
          cat em.log
          cp -p * $SLURM_SUBMIT_DIR

          Usage on Pitzer

          Set-up

          To load the module for the default version of GROMACS, which initializes your environment for the GROMACS application, use module load gromacs.

          Using GROMACS

          To execute a serial GROMACS versions 5 program interactively, simply run it on the command line, e.g.:

          gmx pdb2gmx
          

          Parallel multinode GROMACS versions 5 programs should be run in a batch environment with srun, e.g.:

          srun gmx_mpi_d mdrun
          

          Note that '_mpi' indicates a parallel executable and '_d' indicates a program built with double precision ('_gpu' denotes a GPU executable built with CUDA).  See the module help output for specific versions for more details on executable naming conventions.

          Batch Usage

          When you log into Pitzer you are actually connected to a login node. To  access the compute nodes, you must submit a job to the batch system for execution. Batch jobs can request multiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

          Interactive Batch Session
          For an interactive batch session on Pitzer, one can run the following command:
          sinteractive -A <project-account> -N 1 -n 40 -t 1:00:00
          
          which gives you one node and 40 cores (-N 1 -n 40) with 1 hour (-t 1:00:00). You may adjust the numbers per your need.
          Non-interactive Batch Job (Parallel Run)

          batch script can be created and submitted for a serial, cuda (GPU), or parallel run. You can create the batch script using any text editor in a working directory on the system of your choice. Sample batch scripts and input files for all types of hardware resources are available here:

          ~srb/workshops/compchem/gromacs/
          

          This simple batch script demonstrates some important points:

          #!/bin/bash
          # GROMACS Tutorial for Solvation Study of Spider Toxin Peptide
          # see fwspider_tutor.pdf
          #SBATCH --job-name=fwsinvacuo.pitzer
          #SBATCH --nodes=2 --ntasks-per-node=48
          #SBATCH --account=<project-account>
          
          module load gromacs
          pbsdcp -p 1OMB.pdb em.mdp $TMPDIR
          # Use TMPDIR for best performance.
          cd $TMPDIR
          pdb2gmx -ignh -ff gromos43a1 -f 1OMB.pdb -o fws.gro -p fws.top -water none
          editconf -f fws.gro -d 0.7
          editconf -f out.gro -o fws_ctr.gro -center 2.0715 1.6745 1.914
          grompp -f em.mdp -c fws_ctr.gro -p fws.top -o fws_em.tpr
          srun gmx_mpi mdrun -ntomp 1 -s fws_em.tpr -o fws_em.trr -c fws_ctr.gro -g em.log -e em.edr
          cat em.log
          cp -p * $SLURM_SUBMIT_DIR/

          Further Reading

          Supercomputer: 
          Service: 

          Gaussian

          Gaussian is the most popular general purpose electronic structure program. Recent versions can perform density functional theory, Hartree-Fock, Möller-Plesset, coupled-cluster, and configuration interaction calculations among others. Geometry optimizations, vibrational frequencies, magnetic properties, and solution modeling are available. It performs well as black-box software on closed-shell ground state systems. 

          Availability and Restrictions

          Versions

          Gaussian is available on the Pitzer and Owen Clusters. These versions are currently available at OSC (S means single node serial/parallel and C means CUDA, i.e., GPU enabled):

          Version Owens Pitzer
          g09e01 S  

          g16a03

          S

          S
          g16b01 SC S
          g16c01 SC* SC*
          * Current default version

          You can use module spider gaussian to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access for Academic Users

          Use of Gaussian for academic purposes requires validation. In order to obtain validation, please contact OSC Help for further instruction.

          Publisher/Vendor/Repository and License Type

          Gaussian, commercial

          Usage

          Usage on Owens

          Set-up on Owens

          To load the default version of the Gaussian module which initalizes your environment for Gaussian, use module load gaussian. To select a particular software version, use module load gaussian/version. For example, use module load gaussian/g09e01 to load Gaussian version g09e01 on Owens. 

          Using Gaussian

          To execute Gaussian, simply run the Gaussian binary (g16 or g09) with the input file on the command line:

          g16 < input.com
          

          When the input file is redirected as above ( < ), the output will be standard output; in this form the output can be seen with viewers or editors when the job is running in a batch queue because the batch output file, which captures standard output, is available in the directory from which the job was submitted.  Alternatively, Gaussian can be invoked without file redirection:

          g16 input.com
          

          in which case the output file will be named 'input.log' and its path will be the working directory when the command started; in this form outputs may not be available when the job is running in a batch queue, for example if the working directory was .

          Batch Usage on Owens

          When you log into owens.osc.edu you are logged into a login node. To gain access to the mutiple processors in the computing environment, you must submit your computations to the batch system for execution. Batch jobs can request mutiple processors and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

          Interactive Batch Session
          For an interactive batch session on Owens, one can run the following command:
          sinteractive -A <project-account> -N 1 -n 28 -t 1:00:00
          
          which gives you 28 cores (-N 1 -n 28) with 1 hour (-t 1:00:00). You may adjust the numbers per your need.
          Non-interactive Batch Job (Serial Run)

          batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Sample batch scripts and Gaussian input files are available here:

          /users/appl/srb/workshops/compchem/gaussian/
          

          This simple batch script demonstrates the important points:

          #!/bin/bash
          #SBATCH --job-name=GaussianJob
          #SBATCH --nodes=1 --ntasks-per-node=28
          #SBATCH --time=1:00:00
          #SBATCH --account=<project-account>
          
          cp input.com $TMPDIR
          # Use TMPDIR for best performance.
          cd $TMPDIR
          module load gaussian
          g16 input.com
          cp -p input.log *.chk $SLURM_SUBMIT_DIR
          
          Note: OSC does not have a functional distributed parallel version (LINDA) of Gaussian. Parallelism of Gaussian at OSC is only via shared memory. Consequently, do not request more than one node for Gaussian jobs on OSC's clusters.

          Usage on Pitzer

          Set-up on Pitzer

          To load the default version of the Gaussian module which initalizes your environment for Gaussian, use module load gaussian

          Using Gaussian

          To execute Gaussian, simply run the Gaussian binary (g16 or g09) with the input file on the command line:

          g16 < input.com
          

          When the input file is redirected as above ( < ), the output will be standard output; in this form the output can be seen with viewers or editors when the job is running in a batch queue because the batch output file, which captures standard output, is available in the directory from which the job was submitted.  Alternatively, Gaussian can be invoked without file redirection:

          g16 input.com
          

          in which case the output file will be named 'input.log' and its path will be the working directory when the command started; in this form outputs may not be available when the job is running in a batch queue, for example if the working directory was .

          Batch Usage on Pitzer

          When you log into pitzer.osc.edu you are logged into a login node. To gain access to the mutiple processors in the computing environment, you must submit your computations to the batch system for execution. Batch jobs can request mutiple processors and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

          Interactive Batch Session
          For an interactive batch session on Pitzer, one can run the following command:
          sinteractive -A <project-account> -N 1 -n 40 -t 1:00:00
          
          which gives you 40 cores (-n 40) with 1 hour (-t 1:00:00). You may adjust the numbers per your need.
          Non-interactive Batch Job (Serial Run)

          batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Sample batch scripts and Gaussian input files are available here:

          /users/appl/srb/workshops/compchem/gaussian/
          

          This simple batch script demonstrates the important points:

          #!/bin/bash
          #SBATCH --job-name=GaussianJob
          #SBATCH --nodes=1 --ntasks-per-node=40
          #SBATCH --time=1:00:00
          #SBATCH --account=<project-account>
          
          cp input.com $TMPDIR
          # Use TMPDIR for best performance.
          cd $TMPDIR
          module load gaussian
          g16 input.com
          cp -p input.log *.chk $SLURM_SUBMIT_DIR

          Running Gaussian jobs with GPU

          Gaussian jobs can utilize the P100 GPUS of Owens.  GPUs are not helpful for small jobs but are effective for larger molecules when doing DFT energies, gradients, and frequencies (for both ground and excited states). They are also not used effectively by post-SCF calculations such as MP2 or CCSD. For more

          The above example will utilize CPUs indexed from 0 to 19th, but 0th CPU is associated with 0th GPU.

          A sample batch script for GPU on Owens is as follows:

          #!/bin/bash 
          #SBATCH --job-name=GaussianJob 
          #SBATCH --nodes=1 --ntasks-per-node=40
          #SBATCH --gpus-per-node=1
          #SBATCH --time=1:00:00
          #SBATCH --account=<project-account>
          
          set echo
          cd $TMPDIR
          set INPUT=methane.com
          # SLURM_SUBMIT_DIR refers to the directory from which the job was submitted.
          cp $SLURM_SUBMIT_DIR/$INPUT .
          module load gaussian/g16b01
          g16 < ./$INPUT
          ls -al
          cp -p *.chk $SLURM_SUBMIT_DIR
          

           

          A sample input file for GPU on Owens is as follows:

          %nproc=28
          %mem=8gb
          %CPU=0-27
          %GPUCPU=0=0
          %chk=methane.chk
          #b3lyp/6-31G(d) opt
          methane B3LYP/6-31G(d) opt freq
          0,1
          C        0.000000        0.000000        0.000000
          H        0.000000        0.000000        1.089000
          H        1.026719        0.000000       -0.363000
          H       -0.513360       -0.889165       -0.363000
          H       -0.513360        0.889165       -0.363000

          A sample batch script for GPU on Pitzer is as follows:

          #!/bin/tcsh
          #SBATCH --job-name=methane
          #SBATCH --output=methane.log
          #SBATCH --nodes=1 --ntasks-per-node=48
          #SBATCH --gpus-per-node=1
          #SBATCH --time=1:00:00
          #SBATCH --account=<project-account>
          
          set echo
          cd $TMPDIR
          set INPUT=methane.com
          # SLURM_SUBMIT_DIR refers to the directory from which the job was submitted.
          cp $SLURM_SUBMIT_DIR/$INPUT .
          module load gaussian/g16b01
          g16 < ./$INPUT
          ls -al
          cp -p *.chk $SLURM_SUBMIT_DIR
          

           

          A sample input file for GPU on Pitzer is as follows:

          %nproc=48
          %mem=8gb
          %CPU=0-47
          %GPUCPU=0=0
          %chk=methane.chk
          #b3lyp/6-31G(d) opt
          methane B3LYP/6-31G(d) opt freq
          0,1
          C        0.000000        0.000000        0.000000
          H        0.000000        0.000000        1.089000
          H        1.026719        0.000000       -0.363000
          H       -0.513360       -0.889165       -0.363000
          H       -0.513360        0.889165       -0.363000

           

          Further Reading

          Supercomputer: 
          Service: 

          Git

          Git is a version control system used for tracking file changes and facilitating collaborative work.

          Availability and Restrictions

          Versions

          The following versions of Git are available on OSC clusters:

          Version Owens Pitzer
          2.18.0 X* X*
          * Current default version

          You can use module spider git to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          Git is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Git, Open source

          Usage

          Usage on Owens

          Set-up

          To configure your environment for use of Git, run the following command: module load git. The default version will be loaded. To select a particular Git version, use module load git/version

          Usage on Pitzer

          Set-up

          To configure your environment for use of Git, run the following command: module load git. The default version will be loaded.

          Further Reading

          Supercomputer: 

          Gnuplot

          Gnuplot is a portable command-line driven data and function plotting utility.  It was originally intended to allow scientists and students to visualize mathematical functions and data.  

          Gnuplot supports many types of plots in two or three dimensions.  It can draw using points, lines, boxes, contours, vector fields surfaces and various associated text.  It also supports various specialized plot types.  

          Gnuplot supports many different types of output:  interactive screen display (with mouse and hotkey functionality), pen plotters (like hpgl), printers (including postscript and many color devices), and file formats as vectorial pseudo-devices like LaTeX, metafont, pdf, svg, or bitmap png.  

          Availability and Restrictions

          Versions

          The current versions of Gnuplot available at OSC are:

          Version Owens Pitzer Notes
          4.6 patchlevel 2 System Install   No module needed.
          5.2.2 X*    
          5.2.4   X*  
          * Current default version

          You can use module spider gnuplot to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          Gnuplot is available to all OSC users.

          Publisher/Vendor/Repository and License Type

          Thomas Williams, Colin Kelley/ Open source

          Usage

          Usage on Owens

          To start a Gnuplot session, load the module and launch using the following commands:

          module load gnuplot
          
          gnuplot
          

          To access the Gnuplot help menu, type ? into the Gnuplot command line.  

          Usage on Pitzer

          To start a Gnuplot session, load the module and launch using the following commands:

          module load gnuplot
          
          gnuplot
          

          To access the Gnuplot help menu, type ? into the Gnuplot command line.  

          Further Reading

          For more information, visit the Gnuplot Homepage.  

          Supercomputer: 
          Service: 

          Gurobi

          Gurobi is a mathematical optimization solver that supports a variety of programming and modeling languages.

          Availability and Restrictions

          Versions

          The following versions of bedtools are available on OSC clusters:

          Version Owens
          8.1.1 X*
          * Current default version

          You can use module spider gurobi to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          Gurobi is available to academic OSC users with proper validation. In order to obtain validation, please contact OSC Help for further instruction.

          Publisher/Vendor/Repository and License Type

          Gurobi Optimization, LLC/ Free academic floating license

          Usage

          Usage on Owens

          Set-up

          To configure your environment for use of metilene, run the following command: module load gurobi. The default version will be loaded. To select a particular Gurobi version, use module load gurobi/version. For example, use module load gurobi/8.1.1 to load Gurobi 8.1.1.

          Further Reading

          Supercomputer: 
          Service: 
          Fields of Science: 

          HDF5

          HDF5 is a general purpose library and file format for storing scientific data. HDF5 can store two primary objects: datasets and groups. A dataset is essentially a multidimensional array of data elements, and a group is a structure for organizing objects in an HDF5 file. Using these two basic objects, one can create and store almost any kind of scientific data structure, such as images, arrays of vectors, and structured and unstructured grids.

          Availability and Restrictions

          Versions

          HDF5 is available on the Pitzer and Owens Clusters. The versions currently available at OSC are:

          Version Owens Pitzer Notes
          1.8.17  X    
          1.8.19 X    
          1.10.2 X X  
          1.10.4 X X  
          1.12.0 X* X*  
          * Current Default Version

          You can use module spider hdf5 to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          HDF5 is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          The HDF Group, Open source (academic)

          API Compatibility issue on hdf5/1.12

          hdf5/1.12 may not compatible with applications created with earlier hdf5 versions. In order to work around, users may use a compatibility macro mapping:

          • To compile an application built with a version of HDF5 that includes deprecated symbols (the default), specify: -DH5_USE_110_API (autotools) or –DH5_USE_110_API:BOOL=ON (CMake)

          However, users will not be able to take advantage of some of the new features in 1.12 if using these compatibility mappings. For more detail, please see release note.

          Usage

          Usage on Owens

          Set-up

          Initalizing the system for use of the HDF5 library is dependent on the system you are using and the compiler you are using. To load the default HDF5 library, run the following command: module load hdf5. To load a particular version, use module load hdf5/version. For example, use module load hdf5/1.8.17 to load HDF5 version 1.8.17. You can use module spider hdf5 to view available modules.

          Building With HDF5

          The HDF5 library provides the following variables for use at build time:

          Variable Use
          $HDF5_C_INCLUDE Use during your compilation step for C programs
          $HDF5_CPP_INCLUDE Use during your compilation step for C++ programs (serial version only)
          $HDF5_F90_INCLUDE Use during your compilation step for FORTRAN programs
          $HDF5_C_LIBS Use during your linking step programs
          $HDF5_F90_LIBS

          Use during your linking step for FORTRAN programs

          For example, to build the code myprog.c or myprog.f90 with the hdf5 library you would use:

          icc -c $HDF5_C_INCLUDE myprog.c
          icc -o myprog myprog.o $HDF5_C_LIBS
          ifort -c $HDF5_F90_INCLUDE myprog.f90
          ifort -o myprog myprog.o $HDF5_F90_LIBS
          

          Batch Usage

          When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

          Non-interactive Batch Job (Serial Run)
          A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Below is the example batch script that executes a program built with the HDF5 library:
          #PBS -N AppNameJob
          #PBS -l nodes=1:ppn=28
          module load hdf5
          cd $PBS_O_WORKDIR
          cp foo.dat $TMPDIR
          cd $TMPDIR
          appname
          cp foo_out.h5 $PBS_O_WORKDIR
          

          Usage on Pitzer

          Set-up

          Initalizing the system for use of the HDF5 library is dependent on the system you are using and the compiler you are using. To load the default HDF5 library, run the following command: module load hdf5

          Building With HDF5

          The HDF5 library provides the following variables for use at build time:

          VARIABLE USE
          $HDF5_C_INCLUDE Use during your compilation step for C programs
          $HDF5_CPP_INCLUDE Use during your compilation step for C++ programs (serial version only)
          $HDF5_F90_INCLUDE Use during your compilation step for FORTRAN programs
          $HDF5_C_LIBS Use during your linking step programs
          $HDF5_F90_LIBS

          Use during your linking step for FORTRAN programs

          For example, to build the code myprog.c or myprog.f90 with the hdf5 library you would use:

          icc -c $HDF5_C_INCLUDE myprog.c
          icc -o myprog myprog.o $HDF5_C_LIBS
          ifort -c $HDF5_F90_INCLUDE myprog.f90
          ifort -o myprog myprog.o $HDF5_F90_LIBS
          

          Batch Usage

          When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

          Non-interactive Batch Job (Serial Run)
          batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Below is the example batch script that executes a program built with the HDF5 library:
          #PBS -N AppNameJob
          #PBS -l nodes=1:ppn=28
          module load hdf5
          cd $PBS_O_WORKDIR
          cp foo.dat $TMPDIR
          cd $TMPDIR
          appname
          cp foo_out.h5 $PBS_O_WORKDIR

          Further Reading

          Tag: 
          Supercomputer: 
          Service: 

          HDF5-Serial

          HDF5 is a general purpose library and file format for storing scientific data. HDF5 can store two primary objects: datasets and groups. A dataset is essentially a multidimensional array of data elements, and a group is a structure for organizing objects in an HDF5 file. Using these two basic objects, one can create and store almost any kind of scientific data structure, such as images, arrays of vectors, and structured and unstructured grids.

          For mpi-dependent codes, use the non-serial HDF5 module.

          Availability and Restrictions

          Versions

          HDF5 is available for serial code on Pitzer and Owens Clusters. The versions currently available at OSC are:

          Version Owens Pitzer Notes
          1.8.17 X    
          1.8.19      
          1.10.2 X X  
          1.10.4 X X  
          1.10.5 X X  
          1.12.0 X* X*  
          * Current Default Version

          You can use module spider hdf5-serial to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          HDF5 is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          The HDF Group, Open source (academic)

          Usage

          Usage on Owens

          Set-up

          Initalizing the system for use of the HDF5 library is dependent on the system you are using and the compiler you are using. To load the default serial HDF5 library, run the following command: module load hdf5-serial. To load a particular version, use module load hdf5-serial/version. For example, use module load hdf5-serial/1.10.5 to load HDF5 version 1.10.5. You can use module spider hdf5-serial to view available modules.

          Building With HDF5

          The HDF5 library provides the following variables for use at build time:

          Variable Use
          $HDF5_C_INCLUDE Use during your compilation step for C programs
          $HDF5_CPP_INCLUDE Use during your compilation step for C++ programs (serial version only)
          $HDF5_F90_INCLUDE Use during your compilation step for FORTRAN programs
          $HDF5_C_LIBS Use during your linking step programs
          $HDF5_F90_LIBS

          Use during your linking step for FORTRAN programs

          For example, to build the code myprog.c or myprog.f90 with the hdf5 library you would use:

          icc -c $HDF5_C_INCLUDE myprog.c
          icc -o myprog myprog.o $HDF5_C_LIBS
          ifort -c $HDF5_F90_INCLUDE myprog.f90
          ifort -o myprog myprog.o $HDF5_F90_LIBS
          

          Batch Usage

          When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

          Non-interactive Batch Job (Serial Run)
          A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Below is the example batch script that executes a program built with the HDF5 library:
          #PBS -N AppNameJob
          #PBS -l nodes=1:ppn=28
          module load hdf5
          cd $PBS_O_WORKDIR
          cp foo.dat $TMPDIR
          cd $TMPDIR
          appname
          cp foo_out.h5 $PBS_O_WORKDIR
          

          Usage on Pitzer

          Set-up

          Initalizing the system for use of the HDF5 library is dependent on the system you are using and the compiler you are using. To load the default serial HDF5 library, run the following command: module load hdf5-serial. To load a particular version, use module load hdf5-serial/version. For example, use module load hdf5-serial/1.10.5 to load HDF5 version 1.10.5. You can use module spider hdf5-serial to view available modules.

          Building With HDF5

          The HDF5 library provides the following variables for use at build time:

          VARIABLE USE
          $HDF5_C_INCLUDE Use during your compilation step for C programs
          $HDF5_CPP_INCLUDE Use during your compilation step for C++ programs (serial version only)
          $HDF5_F90_INCLUDE Use during your compilation step for FORTRAN programs
          $HDF5_C_LIBS Use during your linking step programs
          $HDF5_F90_LIBS

          Use during your linking step for FORTRAN programs

          For example, to build the code myprog.c or myprog.f90 with the hdf5 library you would use:

          icc -c $HDF5_C_INCLUDE myprog.c
          icc -o myprog myprog.o $HDF5_C_LIBS
          ifort -c $HDF5_F90_INCLUDE myprog.f90
          ifort -o myprog myprog.o $HDF5_F90_LIBS
          

          Batch Usage

          When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

          Non-interactive Batch Job (Serial Run)
          batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Below is the example batch script that executes a program built with the HDF5 library:
          #PBS -N AppNameJob
          #PBS -l nodes=1:ppn=28
          module load hdf5
          cd $PBS_O_WORKDIR
          cp foo.dat $TMPDIR
          cd $TMPDIR
          appname
          cp foo_out.h5 $PBS_O_WORKDIR

          Further Reading

          Supercomputer: 
          Service: 

          HISAT2

          HISAT2 is a graph-based alignment program that maps DNA and RNA sequencing reads to a population of human genomes.

          Availability and Restrictions

          Versions

          HISAT2 is available on the Owens Cluster. The versions currently available at OSC are:

          Version Owens Pitzer
          2.1.0 X* X*
          * Current Default Version

          You can use module spider hisat2 to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          HISAT2 is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          https://ccb.jhu.edu/software/hisat2, Open source

          Usage

          Usage on Owens and Pitzer

          Set-up

          To configure your enviorment for use of HISAT2, use command module load hisat2. This will load the default version.

          Further Reading

           
          Supercomputer: 
          Fields of Science: 

          HOMER

          HOMER (Hypergeometric Optimization of Motif EnRichment) is a suite of tools for Motif Discovery and ChIP-Seq analysis. It is a collection of command line programs for unix-style operating systems written in mostly perl and c++. Homer was primarily written as a de novo motif discovery algorithm that is well suited for finding 8-12 bp motifs in large scale genomics data.

          Availability and Restrictions

          Versions

          The following versions of HOMER are available on OSC clusters:

          Version Owens
          4.8 X*
          4.10 X
          * Current default version

          You can use  module spider homer to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          HOMER is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Christopher Benner, Open source

          Usage

          Usage on Owens

          Set-up

          To configure your environment for use of HOMER, run the following command: module load homer. The default version will be loaded. To select a particular HOMER version, use module load homer/version. For example, use module load homer/4.10 to load HOMER 4.10.

          Access HOMER Genome Data

          Up-to-date HOMER genome data can be found in $HOMER_DATA/genomes. To the appropriate genome for analyzing genomic motifs, you can specify the path to a file or directory containing the genomic sequence in FASTA format and specify a path for preparsed data:
           
          #!/bin/bash
          #SBATCH --job-name homer_data_test 
          #SBATCH --time=1:00:00
          #SBATCH --nodes=1 --ntasks-per-node=1
          #SBATCH --acount=<project-account>
          
          cp output_test.fastq $TMPDIR
          
          module load homer/4.10
          
          cd $TMPDIR
          homerTools trim -3 GTCTTT -mis 1 -minMatchLength 4 -min 15 output_test.fastq
          
          pbsdcp --gather --recursive --preserve '*' $SLURM_SUBMIT_DIR
           

          Further Reading

          Supercomputer: 
          Service: 
          Fields of Science: 

          HPC Toolkit

          HPC Toolkit is a collection of tools that measure a program's work, resource consumption, and inefficiency to analze performance.

          Availability and Restrictions

          Versions

          The following versions of HPC Toolkitare available on OSC clusters:

          Version Owens Pitzer
          5.3.2 X*  
          2018.09   X*
          * Current default version

          You can use module spider hpctoolkit to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          HPC Toolkit is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Rice Univerity, Open source

          Usage

          Usage on Owens

          Set-up

          To configure your environment for use of HPC Toolkit, run the following command: module load hpctoolkit. The default version will be loaded. To select a particular HPC Toolkit version, use module load hpctoolkit/version

          Usage on Pitzer

          Set-up

          To configure your environment for use of HPC Toolkit, run the following command: module load hpctoolkit. The default version will be loaded. To select a particular HPC Toolkit version, use module load hpctoolkit/version

          Further Reading

          Supercomputer: 

          HTSlib

          HTSlib is a C library used for reading and writing high-throughput sequencing data. HTSlib is the core library used by SAMtools. HTSlib also provides the bgziphtsfile, and tabix utilities.

          Availability and Restrictions

          Versions

          The versions of HTSlib currently available at OSC are:

          Version Owens Pitzer
          1.6 X*  
          1.11   X*
          * Current Default Version

          You can use module spider htslib to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          HTSlib is available to all OSC users.

          Publisher/Vendor/Repository and License Type

          Genome Research Ltd., Open source

          Usage

          Usage on Owens and Pitzer

          Set-up

          To configure your enviorment for use of HTSlib, use command module load htslib. This will load the default version.

          Further Reading

          Supercomputer: 

          Hadoop

          A hadoop cluster can be launched within the HPC environment, but managed by the PBS job scheduler using  Myhadoop framework developed by San Diego Supercomputer Center. (Please see http://www.sdsc.edu/~allans/MyHadoop.pdf)

          Availability and Restrictions

          Versions

          The following versions of Hadoop are available on OSC systems: 

          Version Owens
          3.0.0-alpha1 X*
          * Current default version

          You can use module spider hadoop to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          Hadoop is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Apache software foundation, Open source

          Usage

          Set-up

          In order to configure your environment for the usage of Hadoop, run the following command:

          module load hadoop

          In order to access a particular version of Hadoop, run the following command

          module load hadoop/3.0.0-alpha1
          

          Using Hadoop

          In order to run Hadoop in batch, reference the example batch script below. This script requests 6 node on the Owens cluster for 1 hour of walltime. 

          #PBS -N hadoop-example
          
          #PBS -l nodes=6:ppn=12
          
          #PBS -l walltime=01:00:00
          
          setenv WORK $PBS_O_WORKDIR
          
          module load hadoop/3.0.0-alpha1
          
          module load myhadoop/v0.40
          
          setenv HADOOP_CONF_DIR $TMPDIR/mycluster-conf-$PBS_JOBID
          
          cd $TMPDIR
          
          myhadoop-configure.sh -c $HADOOP_CONF_DIR -s $TMPDIR
          
          $HADOOP_HOME/sbin/start-dfs.sh
          
          hadoop dfsadmin -report
          
          hadoop  dfs -mkdir data
          
          hadoop  dfs -put $HADOOP_HOME/README.txt  data/
          
          hadoop  dfs -ls data
          
          hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-alpha1.jar wordcount data/README.txt wordcount-out
          
          hadoop  dfs -ls wordcount-out
          
          hadoop  dfs  -copyToLocal -f  wordcount-out  $WORK
          
          $HADOOP_HOME/sbin/stop-dfs.sh
          
          myhadoop-cleanup.sh
          
          

          Example Jobs

          Please check /usr/local/src/hadoop/3.0.0-alpha1/test.osc folder for more examples of hadoop jobs

          Further Reading

          See Also

          Supercomputer: 
          Service: 
          Fields of Science: 

          Horovod

          "Horovod is a distributed training framework for TensorFlow, Keras, PyTorch, and MXNet. The goal of Horovod is to make distributed Deep Learning fast and easy to use. The primary motivation for this project is to make it easy to take a single-GPU TensorFlow program and successfully train it on many GPUs faster."

          Quote from Horovod Github documentation

          Installation

          Please follow the link for general instructions on installing Horovod for use with GPUs. The commands below assume a Bourne type shell; if you are using a C type shell then the "source activate" command may not work; in general, you can load all the modules, define any environment variables, and then type "bash" and execute the other commands.

          Step 1: Install NCCL 2

          Please download NCCL 2 from https://developer.nvidia.com/nccl (select OS agnostic local installer; Download NCCL 2.7.8, for CUDA 10.2, July 24,2020 was used in the latest test of this recipe).

          Add the nccl library path to LD_LIBRARY_PATH environment variable

          $ export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:Path_to_nccl/nccl-<version>/lib
          Step 2: Install horovod python package
          module load python/3.6-conda5.2

          Create a local python environment for a horovod installation with nccl and activate it

          conda create -n horovod-withnccl python=3.6 anaconda
          source activate horovod-withnccl

          Install a GPU version of tensorflow or pytorch

          pip install https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.10.0-cp36-cp36m-linux_x86_64.whl

          Load mvapich2 and cuda modules

          module load gnu/7.3.0  mvapich2-gdr/2.3.4 
          
          module load cuda/10.2.89

          Install the horovod python package

          HOROVOD_NCCL_HOME=/path_to_nccl_home/ HOROVOD_GPU_ALLREDUCE=NCCL pip install --no-cache-dir horovod

          Testing

          Please get the benchmark script from here.

          #!/bin/bash 
          #SBATCH --job-name R_ExampleJob 
          #SBATCH --nodes=2 --ntasks-per-node=48 
          #SBATCH --time=01:00:00 
          
          
          module load python/3.6-conda5.2 
          module load cuda/10.2.89 
          module load gnu/7.3.0 
          module load mvapich2-gdr/2.3.4 
          source activate horovod-withnccl export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/path_to_nccl_home/lib mpiexec -ppn 1 -binding none -env NCCL_DEBUG=INFO python tf_cnn_benchmarks.py
          

           

          Feel free to contact OSC Help if you have any issues with installation.

          Publisher/Vendor/Repository and License Type

          https://eng.uber.com/horovod/, Open source

          Further Reading

          TensorFlow homepage

          Supercomputer: 
          Service: 
          Technologies: 
          Fields of Science: 

          Intel Compilers

          The Intel compilers for both C/C++ and FORTRAN.

          Availability and Restrictions

          Versions

          The versions currently available at OSC are:

          Version Owens Pitzer Notes
          16.0.3 X    
          16.0.8 X   Security update
          17.0.2 X    
          17.0.5 X    
          17.0.7 X X Security update
          18.0.0 X    
          18.0.2 X    
          18.0.3 X X  
          18.0.4   X  
          19.0.3 X X  
          19.0.5 X* X*  
          * Current Default Version

          You can use module spider intel  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          The Intel Compilers are available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Intel, Commercial (state-wide)

          Usage

          Usage on Owens

          Set-up on Owens

          After you ssh to Owens, the default version of Intel compilers will be loaded for you automatically. 

          Using the Intel Compilers

          Once the intel compiler module has been loaded, the compilers are available for your use. See our compilation guide for suggestions on how to compile your software on our systems. The following table lists common compiler options available in all languages.

          COMPILER OPTION PURPOSE
          -c Compile only; do not link  
          -DMACRO[=value] Defines preprocessor macro MACRO with optional value (default value is 1)  
          -g  Enables debugging; disables optimization  
          -I/directory/name Add /directory/name to the list of directories to be searched for #include files  
          -L/directory/name Adds /directory/name to the list of directories to be searched for library files  
          -lname Adds the library libname.a or libname.so to the list of libraries to be linked  
          -o outfile Names the resulting executable outfile instead of a.out  
          -UMACRO Removes definition of MACRO from preprocessor  
          -v Emit version including gcc compatibility; see below  
            Optimization Options
          -O0 Disable optimization  
          -O1 Light optimization  
          -O2 Heavy optimization (default)  
          -O3 Aggressive optimization; may change numerical results  
          -ipo Inline function expansion for calls to procedures defined in separate files  
          -funroll-loops Loop unrolling  
          -parallel Automatic parallelization  
          -openmp Enables translation of OpenMP directives  

          The following table lists some options specific to C/C++

          -strict-ansi Enforces strict ANSI C/C++ compliance
          -ansi Enforces loose ANSI C/C++ compliance
          -std=val Conform to a specific language standard

          The following table lists some options specific to Fortran

          -convert big_endian Use unformatted I/O compatible with Sun and SGI systems
          -convert cray Use unformatted I/O compatible with Cray systems
          -i8 Makes 8-byte INTEGERs the default
          -module /dir/name Adds /dir/name to the list of directories searched for Fortran 90 modules
          -r8 Makes 8-byte REALs the default
          -fp-model strict Disables optimizations that can change the results of floating point calculations

          Intel compilers use the GNU tools on the clusters:  header files, libraries, and linker.  This is called the Intel and GNU compatibility and interoperability.  Use the Intel compiler option -v to see the gcc version that is currently specified.  Most users will not have to change this.  However, the gcc version can be controlled by users in several ways. 

          On OSC clusters the default mechanism of control is based on modules.  The most noticeable aspect of interoperability is that some parts of some C++ standards are available by default in various versions of the Intel compilers; other parts require you to load an extra module.  The C++ standard can be specified with the Intel compiler option -std=val; see the compiler man page for valid values of val.  If you specify a particular standard then load the corresponding module; the most common Intel compiler version and C++ standard combinations, that are applicable to this cluster, are described below:

          For the C++14 standard with an Intel 16 compiler:

          module load cxx14
          

          With an Intel 17 or 18 compiler, module cxx17 will be automatically loaded by the intel module load command to enable the GNU tools necessary for the C++17 standard.   With an Intel 19 compiler, module gcc-compatibility will be automatically loaded by the intel module load command to enable the GNU tools necessary for the C++17 standard.  (In early 2020 OSC changed the name of these GNU tool controlling modules to clarify their purpose and because our underlying implementation changed.)

          A symptom of broken gcc-compatibility is unusual or non sequitur compiler errors typically involving the C++ standard library especially with respect to template instantiation, for example:

              error: more than one instance of overloaded function "std::to_string" matches the argument list:
                        detected during:
                          instantiation of "..."
          
              error: class "std::vector<std::pair<short, short>, std::allocator<std::pair <short, short>>>" has no member "..."
                        detected during:
                          instantiation of "..."
          

          An alternative way to control compatibility and interoperability is with Intel compiler options; see the "GNU gcc Interoperability" sections of the various Intel compiler man pages for details.

          Batch Usage on Owens

          When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

          Interactive Batch Session
          For an interactive batch session on Owens, one can run the following command:
          sinteractive -A <project-account> -N 1 -n 28 -t 1:00:00
          
          which gives you 1 node with 28 cores ( -N 1 -n 28 ) with 1 hour ( -t 1:00:00 ). You may adjust the numbers per your need.
          Non-interactive Batch Job (Serial Run)
          batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will use the input file named  hello.c  and the output file named  hello_results . Below is the example batch script ( job.txt ) for a serial run:
          #!/bin/bash
          #SBATCH --time=1:00:00 
          #SBATCH --nodes=1 --ntasks-per-node=28 
          #SBATCH --job-name jobname 
          #SBATCH --account=<project-account>
          
          module load intel 
          cp hello.c $TMPDIR 
          cd $TMPDIR 
          icc -O2 hello.c -o hello 
          ./hello > hello_results 
          cp hello_results $SLURM_SUBMIT_DIR
          

          In order to run it via the batch system, submit the  job.txt  file with the following command:

          sbatch job.txt
          
          Non-interactive Batch Job (Parallel Run)
          Below is the example batch script ( job.txt ) for a parallel run:
          #!/bin/bash
          #SBATCH --time=1:00:00
          #SBATCH--nodes=2 --ntasks-per-node=40
          #SBATCH --job-name name
          #SBATCH --account=<project-account>
          
          module load intel
          mpicc -O2 hello.c -o hello
          cp hello $TMPDIR
          cd $TMPDIR
          mpiexec ./hello > hello_results
          cp hello_results $SLURM_SUBMIT_DIR
          

          Usage on Pitzer

          Set-up on Pitzer

          After you ssh to Pitzer, the default version of Intel compilers will be loaded for you automatically. 

          Using the Intel Compilers

          Once the intel compiler module has been loaded, the compilers are available for your use. See our compilation guide for suggestions on how to compile your software on our systems. The following table lists common compiler options available in all languages.

          COMPILER OPTION PURPOSE
          -c Compile only; do not link  
          -DMACRO[=value] Defines preprocessor macro MACRO with optional value (default value is 1)  
          -g  Enables debugging; disables optimization  
          -I/directory/name Add /directory/name to the list of directories to be searched for #include files  
          -L/directory/name Adds /directory/name to the list of directories to be searched for library files  
          -lname Adds the library libname.a or libname.so to the list of libraries to be linked  
          -o outfile Names the resulting executable outfile instead of a.out  
          -UMACRO Removes definition of MACRO from preprocessor  
          -v Emit version including gcc compatibility; see below
            Optimization Options
          -O0 Disable optimization  
          -O1 Light optimization  
          -O2 Heavy optimization (default)  
          -O3 Aggressive optimization; may change numerical results  
          -ipo Inline function expansion for calls to procedures defined in separate files  
          -funroll-loops Loop unrolling  
          -parallel Automatic parallelization  
          -openmp Enables translation of OpenMP directives  

          The following table lists some options specific to C/C++

          -strict-ansi Enforces strict ANSI C/C++ compliance
          -ansi Enforces loose ANSI C/C++ compliance
          -std=val Conform to a specific language standard

          The following table lists some options specific to Fortran

          -convert big_endian Use unformatted I/O compatible with Sun and SGI systems
          -convert cray Use unformatted I/O compatible with Cray systems
          -i8 Makes 8-byte INTEGERs the default
          -module /dir/name Adds /dir/name to the list of directories searched for Fortran 90 modules
          -r8 Makes 8-byte REALs the default
          -fp-model strict Disables optimizations that can change the results of floating point calculations

          Intel compilers use the GNU tools on the clusters:  header files, libraries, and linker.  This is called the Intel and GNU compatibility and interoperability.  Use the Intel compiler option -v to see the gcc version that is currently specified.  Most users will not have to change this.  However, the gcc version can be controlled by users in several ways. 

          On OSC clusters the default mechanism of control is based on modules.  The most noticeable aspect of interoperability is that some parts of some C++ standards are available by default in various versions of the Intel compilers; other parts require an extra module.  The C++ standard can be specified with the Intel compiler option -std=val; see the compiler man page for valid values of val.

          With an Intel 17 or 18 compiler, module cxx17 will be automatically loaded by the intel module load command to enable the GNU tools necessary for the C++17 standard.   With an Intel 19 compiler, module gcc-compatibility will be automatically loaded by the intel module load command to enable the GNU tools necessary for the C++17 standard.  (In early 2020 OSC changed the name of these GNU tool controlling modules to clarify their purpose and because our underlying implementation changed.)

          A symptom of broken gcc-compatibility is unusual or non sequitur compiler errors typically involving the C++ standard library especially with respect to template instantiation, for example:

              error: more than one instance of overloaded function "std::to_string" matches the argument list:
                        detected during:
                          instantiation of "..."
          
              error: class "std::vector<std::pair<short, short>, std::allocator<std::pair <short, short>>>" has no member "..."
                        detected during:
                          instantiation of "..."
          

          An alternative way to control compatibility and interoperability is with Intel compiler options; see the "GNU gcc Interoperability" sections of the various Intel compiler man pages for details.

           

          C++ Standard GNU Intel
          C++11 > 4.8.1 > 14.0
          C++14 > 6.1 > 17.0
          C++17 > 7 > 19.0
          C++2a features available since 8  

           

          Batch Usage on Pitzer

          When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

          Interactive Batch Session

          For an interactive batch session on Pitzer, one can run the following command:

          sinteractive -A <project-account> -N 1 -n 40 -t 1:00:00
          

          which gives you 1 node (-N 1), 40 cores ( -n 40), and 1 hour ( -t 1:00:00). You may adjust the numbers per your need.

          Non-interactive Batch Job (Serial Run)

          batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will use the input file named  hello.c  and the output file named  hello_results . Below is the example batch script ( job.txt ) for a serial run:

          #!/bin/bash
          #SBATCH --time=1:00:00
          #SBATCH --nodes=1 --ntasks-per-node=40
          #SBATCH --job-name hello
          #SBATCH --account=<project-account>
          
          module load intel
          cp hello.c $TMPDIR
          cd $TMPDIR
          icc -O2 hello.c -o hello
          ./hello > hello_results
          cp hello_results $SLURM_SUBMIT_DIR
          

          In order to run it via the batch system, submit the   job.txt  file with the following command:

          sbatch job.txt
          
          Non-interactive Batch Job (Parallel Run)

          Below is the example batch script ( job.txt ) for a parallel run:

          #!/bin/bash
          #SBATCH --time=1:00:00
          #SBATCH --nodes=2 --ntasks-per-node=40
          #SBATCH --job-name name
          #SBATCH --account=<project-account>
          
          module load intel
          module laod intelmpi
          mpicc -O2 hello.c -o hello
          cp hello $TMPDIR
          cd $TMPDIR
          sun ./hello > hello_results
          cp hello_results $SLURM_SUBMIT_DIR
          

          Further Reading

          See Also

          Supercomputer: 
          Service: 
          Technologies: 
          Fields of Science: 

          Intel MPI

          Intel's implementation of the Message Passing Interface (MPI) library. See Intel Compilers for available compiler versions at OSC.

          Availability and Restrictions

          Versions

          Intel MPI may be used as an alternative to - but not in conjunction with - the MVAPICH2 MPI libraries. The versions currently available at OSC are:

          Version Owens Pitzer
          5.1.3 X  
          2017.2 X  
          2017.4 X X
          2018.0 X  
          2018.3 X X
          2018.4   X
          2019.3 X X
          2019.7 X* X*
          * Current Default Version

          You can use module spider intelmpi to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          Intel MPI is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Intel, Commercial

          Known Software Issues

          A partial-node MPI job failed to start using mpiexec

          Update: October 2020
          Version: 2019.3 2019.7

          A partial-node MPI job may fail to start using mpiexec from intelmpi/2019.3 and intelmpi/2019.7 with error messages like

          [mpiexec@o0439.ten.osc.edu] wait_proxies_to_terminate (../../../../../src/pm/i_hydra/mpiexec/intel/i_mpiexec.c:532): downstream from host o0439 was killed by signal 11 (Segmentation fault)
          [mpiexec@o0439.ten.osc.edu] main (../../../../../src/pm/i_hydra/mpiexec/mpiexec.c:2114): assert (exitcodes != NULL) failed
          
          /var/spool/torque/mom_priv/jobs/11510761.owens-batch.ten.osc.edu.SC: line 30: 11728 Segmentation fault  
          
          /var/spool/slurmd/job00884/slurm_script: line 24:  3180 Segmentation fault      (core dumped)
          

          If you are using SLURM, make sure the job has CPU resource allocation using #SBATCH --ntasks=N instead of

          #SBATCH --nodes=1
          #SBATCH --ntasks-per-node=N
          

          If you are using PBS, please use Intel MPI 2018 or intelmpi/2019.3 with the module libfabric/1.8.1.

          Using mpiexec with SLURM

          Update: October 2020
          Version: 2017.x 2018.x 2019.x
          Intel MPI on SLURM batch system is configured to support PMI and Hydra process managers. It is recommended to use srun as MPI program launcher. If you prefer using mpiexec with SLURM, you might experience MPI init error or see a warning:
          MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
          Please set unset I_MPI_PMI_LIBRARY in a batch job script before running MPI programs to resolve the issue.

          MPI-IO issues on home directories

          Update: May 2020
          Version: 2019.3
          Certain MPI-IO operations with intelmpi/2019.3 may crash, fail or proceed with errors on the home directory. We do not expect the same issue on our GPFS file system, such as the project space and the scratch space. The problem might be related to the known issue reported by HDF5 group. Please read the section "Problem Reading A Collectively Written Dataset in Parallel" from HDF5 Known Issues for more detail.

          Usage

          Usage on Owens

          Set-up on Owens

          To configure your environment for the default version of Intel MPI, use module load intelmpi. To configure your environment for a specific version of Intel MPI, use module load intelmpi/version. For example, use module load intelmpi/5.1.3 to load Intel MPI version 5.1.3 on Owens.

          You can use module spider intelmpi to view available modules on Owens.

          Note: This module conflicts with the default loaded MVAPICH installations, and Lmod will automatically replace with the correct one when you use module load intelmpi.

          Using Intel MPI

          Software compiled against this module will use the libraries at runtime.

          Building With Intel MPI

          On Ruby, we have defined several environment variables to make it easier to build and link with the Intel MPI libraries.

          VARIABLE USE
          $MPI_CFLAGS Use during your compilation step for C programs.
          $MPI_CXXFLAGS Use during your compilation step for C++ programs.
          $MPI_FFLAGS Use during your compilation step for Fortran programs.
          $MPI_F90FLAGS Use during your compilation step for Fortran 90 programs.
          $MPI_LIBS Use when linking your program to Intel MPI.

          In general, for any application already set up to use mpicc, (or similar), compilation should be fairly straightforward.

          Batch Usage on Owens

          When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the multiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request multiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.

          Non-interactive Batch Job (Parallel Run)
          A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will run a program compiled against Intel MPI (called my-impi-application) for five hours on Owens:
          #!/bin/bash
          #SBATCH --job-name MyIntelMPIJob
          #SBATCH --nodes=4 --ntasks-per-node=28
          #SBATCH --time=5:00:00
          #SBATCH --account=<project-account>
          
          module swap mvapich2 intelmpi
          mpiexec my-impi-application
          

          Usage on Pitzer

          Set-up on Pitzer

          To configure your environment for the default version of Intel MPI, use module load intelmpi.
          Note: This module conflicts with the default loaded MVAPICH installations, and Lmod will automatically replace with the correct one when you use module load intelmpi.

          Using Intel MPI

          Software compiled against this module will use the libraries at runtime.

          Building With Intel MPI

          On Oakley, we have defined several environment variables to make it easier to build and link with the Intel MPI libraries.

          VARIABLE USE
          $MPI_CFLAGS Use during your compilation step for C programs.
          $MPI_CXXFLAGS Use during your compilation step for C++ programs.
          $MPI_FFLAGS Use during your compilation step for Fortran programs.
          $MPI_F90FLAGS Use during your compilation step for Fortran 90 programs.
          $MPI_LIBS Use when linking your program to Intel MPI.

          In general, for any application already set up to use mpicc compilation should be fairly straightforward.

          Batch Usage on Pitzer

          When you log into pitzer.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the multiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request multiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.

          Non-interactive Batch Job (Parallel Run)
          A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will run a program compiled against Intel MPI (called my-impi-application) for five hours on Pitzer:
          #!/bin/bash
          #SBATCH --job-name MyIntelMPIJob
          #SBATCH --nodes=2 --ntasks-per-node=48
          #SBATCH --time=5:00:00
          #SBATCH --account=<project-account>
          
          module load intelmpi
          srun my-impi-application

          Further Reading

          See Also

          Java

          Java is a concurrent, class-based, object-oriented programming language.

          Availability and Restrictions

          Versions

          The following versions of Java are available on OSC clusters:

          Version Owens Pitzer
          1.7.0 X  
          1.8.0_131 X* X*
          * Current default version

          You can use module spider java to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          Java is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Oracle, Freeware

          Usage

          Usage on Owens

          Set-up

          To configure your environment for use of Java, run the following command: module load java. The default version will be loaded. To select a particular Java version, use module load java/version

          Usage on Pitzer

          Set-up

          To configure your environment for use of Java, run the following command: module load java. The default version will be loaded. To select a particular Java version, use module load java/version

          Further Reading

          Supercomputer: 

          Julia

          From julialang.org:

          "Julia is a high-level, high-performance dynamic programming language for numerical computing. It provides a sophisticated compiler, distributed parallel execution, numerical accuracy, and an extensive mathematical function library. Julia’s Base library, largely written in Julia itself, also integrates mature, best-of-breed open source C and Fortran libraries for linear algebra, random number generation, signal processing, and string processing. In addition, the Julia developer community is contributing a number of external packages through Julia’s built-in package manager at a rapid pace. IJulia, a collaboration between the Jupyter and Julia communities, provides a powerful browser-based graphical notebook interface to Julia."

          Availability and Restrictions

          Versions

          Julia is available on Owens Clusters. The versions currently available at OSC are:

          Version Owens Pitzer Notes
          0.5.1  X*    
          0.6.4 X    
          1.0.0 X X*  
          1.1.1 X X  
          1.3.1 X X  
          *:Current default version

          You can use module spider julia to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          Julia is available for use by all OSC users.

          Publisher/Vendor/Repository and License Type

          Jeff Bezanson et al., Open source

          Usage 

          Interactive Julia Notebooks

          If you are using OnDemand, you can simply work with Jupyter and the selection of the Julia kernel to use interactive notebooks to work on an Owens or Pitzer compute node!

          Navigate to ondemand.osc.edu and select a Jupyter notebook:

          Jupyter Notebook

          Since version 1.0, OSC user must manage own IJulia kernels in Jupyter notebooks. The following is an example of adding the latest version of IJulia to Julia 1.0.0:

          $ module load julia/1.0.0
          $ julia
          julia> ]
          (v1.0) pkg> add IJulia
          [ .. installation output .. ]
          (v1.0) pkg> st IJulia
              Status `~/.julia/environments/v1.0/Project.toml`
            [7073ff75] IJulia v1.20.0
          

          Then on Juptyer app, you can find the item User Defined Julia 1.0.0 in the kernel drop-down menu:

          Screen Shot 2019-11-07 at 12.04.22 PM.png

          For more detail about package management, please refer to the Julia document

          Supercomputer: 
          Service: 
          Fields of Science: 

          Kallisto

          Kallisto is an RNA-seq quantification program. It quantifies abundances of transcripts from RNA-seq data and uses psedoalignment to determine the compatibility of reads with targets, without needing alignment.

          Availability and Restrictions

          Versions

          Kallisto is available on the Owens Clusters. The versions currently available at OSC are:

          Version Owens
          0.43.1 X*
          * Current Default Version

          You can use module spider kallisto to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          Kallisto is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Nicolas Bray et al., Open source

          Usage

          Usage on Owens

          Set-up

          To configure your enviorment for use of Salmon, use command module load kallisto. This will load the default version.

          Further Reading

          Supercomputer: 
          Fields of Science: 

          LAMMPS

          The Large-scale Atomic/Molecular Massively Parallel Simulator (LAMMPS) is a classical molecular dynamics code designed for high-performance simulation of large atomistic systems.  LAMMPS generally scales well on OSC platforms, provides a variety of modeling techniques, and offers GPU accelerated computation.

          Availability and Restrictions

          Versions

          LAMMPS is available on all clusters. The following versions are currently installed at OSC:

          Version Owens Pitzer
          14May16 P  
          31Mar17 PC  
          16Mar18 PC  
          22Aug18 PC PC
          5Jun19 PC PC
          3Mar20 PC* PC*
          29Oct20 PC* PC*
          * Current default version; S = serial executables; P = parallel; C = CUDA
          *  IMPORTANT NOTE: You must load the correct compiler and MPI modules before you can load LAMMPS. To determine which modules you need, use module spider lammps/{version}.  Some LAMMPS versions are available with multiple compiler versions and MPI versions; in general, we recommend using the latest versions. (In particular, mvapich2/2.3.2 is recommended over 2.3.1 and 2.3; see the known issue.

          You can use module spider lammps  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          LAMMPS is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Sandia National Lab., Open source

          Usage

          Usage on Owens

          Set-up

          To load the default version of LAMMPS module and set up your environment, use  module load lammps . To select a particular software version, use module load lammps/version . For example, use  module load lammps/14May16  to load LAMMPS version 14May16. 

          Using LAMMPS

          Once a module is loaded, LAMMPS can be run with the following command:
          lammps < input.file
          

          To see information on the packages and executables for a particular installation, run the module help command, for example:

          module help lammps
          

          Batch Usage

          By connecting to owens.osc.edu you are logged into one of the login nodes which has computing resource limits. To gain access to the manifold resources on the cluster, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

          Interactive Batch Session

          For an interactive batch session one can run the following command:

          sinteractive -A <project-account> -N 1 -n 28 -g 1 -t 00:20:00 
          

          which requests one whole node with 28 cores ( -N 1 -n 28), for a walltime of 20 minutes ( -t 00:20:00 ), with one gpu (-g 1). You may adjust the numbers per your need.

          Non-interactive Batch Job (Parallel Run)

          A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Sample batch scripts and LAMMPS input files are available here:

          ~srb/workshops/compchem/lammps/
          

          Below is a sample batch script. It asks for 56 processors and 10 hours of walltime. If the job goes beyond 10 hours, the job would be terminated.

          #!/bin/bash
          #SBATCH --job-name=chain  
          #SBATCH --nodes=2 --ntasks-per-node=28  
          #SBATCH --time=10:00:00  
          #SBATCH --account=<project-account>
          
          module load lammps  
          pbsdcp chain.in $TMPDIR  
          cd $TMPDIR  
          lammps < chain.in  
          pbsdcp -g * $SLURM_SUBMIT_DIR
          

          Usage on Pitzer

          Set-up

          To load the default version of LAMMPS module and set up your environment, use  module load lammps

          Using LAMMPS

          Once a module is loaded, LAMMPS can be run with the following command:
          lammps < input.file
          

          To see information on the packages and executables for a particular installation, run the module help command, for example:

          module help lammps
          

          Batch Usage

          To access a cluster's main computational resources, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

          Interactive Batch Session

          For an interactive batch session one can run the following command:

          sinteractive -A <project-account> -N 1 -n 48 -g 1 -t 00:20:00 
          

          which requests one whole node with 28 cores ( -N 1 -n 48), for a walltime of 20 minutes ( -t 00:20:00 ), with one gpu (-g 1). You may adjust the numbers per your need.

          Non-interactive Batch Job (Parallel Run)

          batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Sample batch scripts and LAMMPS input files are available here:

          ~srb/workshops/compchem/lammps/
          

          Below is a sample batch script. It asks for 56 processors and 10 hours of walltime. If the job goes beyond 10 hours, the job would be terminated.

          #!/bin/bash
          #SBATCH --job-name=chain 
          #SBATCH --nodes=2 --ntasks-per-node=48 
          #SBATCH --time=10:00:00 
          #SBATCH --account=<project-account>
          
          module load lammps 
          pbsdcp chain.in $TMPDIR 
          cd $TMPDIR 
          lammps < chain.in 
          pbsdcp -g * $SLURM_SUBMIT_DIR

          Further Reading

          Supercomputer: 
          Service: 

          LAPACK

          LAPACK (Linear Algebra PACKage) provides routines for solving systems of simultaneous linear equations, least-squares solutions of linear systems of equations, eigenvalue problems, and singular value problems.

          Availability and Restrictions

          A highly optimized implementation of LAPACK is available on all OSC clusters as part of the Intel Math Kernel Library (MKL). We recommend that you use MKL rather than building LAPACK for yourself. MKL is available to all OSC users.

          Publisher/Vendor/Repository and License Type

          http://www.netlib.org/lapack/, Open source

          Usage

          See OSC's MKL software page for usage information. Note that there are lapack shared libraries on the clusters; however, these are old versions from the operating system and should generally not be used.  You should modify your makefile or build script to link to the MKL libraries instead; a quick start for a crude approach is to merely load an mkl module and substitute the consequently defined environment variable $(MKL_LIBS) for -llapack.

          Further Reading

          Supercomputer: 
          Service: 
          Technologies: 
          Fields of Science: 

          LS-DYNA

          LS-DYNA is a general purpose finite element code for simulating complex structural problems, specializing in nonlinear, transient dynamic problems using explicit integration. LS-DYNA is one of the codes developed at Livermore Software Technology Corporation (LSTC).

          Availability and Restrictions

          Versions

          LS-DYNA is available on Owens and Oakley Clusters for both serial (smp solver for single node jobs) and parallel (mpp solver for multipe node jobs) versions. The versions currently available at OSC are:

          Version Owens

          9.0.1

          smp  X
          mpp X
          10.1.0 smp X
          mpp X
          11.0.0 smp X*
          mpp X*
          * Current default version

          You can use module spider ls-dyna to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access for Academic Users

          ls-dyna is available to academic OSC users with proper validation. In order to obtain validation, please contact OSC Help for further instruction.

          Access for Commerical Users

          Contact OSC Help for getting access to LS-DYNA if you are a commerical user.

          Publisher/Vendor/Repository and License Type

          LSTC, Commercial

          Usage

          Usage on Owens

          Set-up on Owens

          To view available modules installed on Owens, use  module spider ls-dyna for smp solvers, and use  module spider mpp for mpp solvers. In the module name, '_s' indicates single precision and '_d' indicates double precision. For example, mpp-dyna/971_d_9.0.1 is the mpp solver with double precision on Owens. Use  module load name to load LS-DYNA with a particular software version. For example, use  module load mpp-dyna/971_d_9.0.1 to load LS-DYNA mpp solver version 9.0.1 with double precision on Owens.

          Batch Usage on Owens

          When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

          Interactive Batch Session

          For an interactive batch session one can run the following command:

          qsub -I -l nodes=1:ppn=28 -l walltime=00:20:00 
          
          which requests one whole node with 28 cores (-l nodes=1:ppn=28), for a walltime of 20 minutes (-l walltime=00:20:00). You may adjust the numbers per your need.
          Non-interactive Batch Job (Serial Run)

          batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Please follow the steps below to use LS-DYNA via the batch system:

          1) copy your input files (explorer.k in the example below) to your work directory at OSC

          2) create a batch script, similar to the following file, saved as job.txt. It uses the smp solver for a serial job (nodes=1) on Owens:

          #PBS -N plate_test
          #PBS -l walltime=5:00:00
          #PBS -l nodes=1:ppn=28
          #PBS -j oe
          # The following lines set up the LSDYNA environment
          module load ls-dyna/971_d_9.0.1
          #
          # Move to the directory where the input files are located
          #
          cd $PBS_O_WORKDIR
          #
          # Run LSDYNA (number of cpus > 1)
          #
          lsdyna I=explorer.k NCPU=28
          

           3) submit the script to the batch queue with the command: qsub job.txt.

           When the job is finished, all the result files will be found in the directory where you submitted your job ($PBS_O_WORKDIR). Alternatively, you can submit your job from the temporary directory ($TMPDIR), which is faster to access for the system and might be beneficial for bigger jobs. Note that $TMPDIR is uniquely associated with the job submitted and will be cleared when the job ends. So you need to copy your results back to your work directory at the end of your script. 

          Non-interactive Batch Job (Parallel Run)
          Please follow the steps below to use LS-DYNA via the batch system:

          1) copy your input files (explorer.k in the example below) to your work directory at OSC

          2) create a batch script, similar to the following file, saved as job.txt). It uses the mmp solver for a parallel job (nodes>1) on Owens:

          #PBS -N plate_test
          #PBS -l walltime=5:00:00
          #PBS -l nodes=2:ppn=28
          #PBS -j oe
          # The following lines set up the LSDYNA environment
          module load intel/18.0.3
          module load intelmpi/2018.3
          module load mpp-dyna/971_d_9.0.1
          #
          # Move to the directory where the input files are located
          #
          cd $PBS_O_WORKDIR
          #
          # Run LSDYNA (number of cpus > 1)
          #
          mpiexec mpp971 I=explorer.k NCPU=56
          

           3) submit the script to the batch queue with the command: qsub job.txt.

          When the job is finished, all the result files will be found in the directory where you submitted your job ($PBS_O_WORKDIR). Alternatively, you can submit your job from the temporary directory ($TMPDIR), which is faster to access for the system and might be beneficial for bigger jobs. Note that $TMPDIR is uniquely associated with the job submitted and will be cleared when the job ends. So you need to copy your results back to your work directory at the end of your script. An example scrip should include the following lines:

          ...
          cd $TMPDIR
          cp $PBS_O_WORKDIR/explorer.k .
          ... #launch the solver and execute
          pbsdcp -g '*' $PBS_O_WORKDIR
          

          Further Reading

          See Also

          Supercomputer: 
          Service: 

          LS-OPT

          LS-OPT is a package for design optimization, system identification, and probabilistic analysis with an interface to LS-DYNA.

          Availability and Restrictions

          Versions

          The following versions of ls-opt are available on OSC clusters:

          Version Owens
          6.0.0 X*
          * Current default version

          You can use module spider ls-opt to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          In order to use LS-OPT, you need LS-DYNA. OSC does not provide LS-DYNA license directly, however users with their own academic departmental license can use it on the OSC clusters.  Please contact OSC Help for further instruction.

          Publisher/Vendor/Repository and License Type

          LSTC, Commercial

          Usage

          Usage on Owens

          Set-up

          To configure your environment for use of LS-OPT, run the following command: module load ls-opt. The default version will be loaded. To select a particular LS-OPT version, use module load ls-opt/version. For example, use module load ls-opt/6.0.0 to load LS-OPT 6.0.0.

          Further Reading

          Supercomputer: 
          Service: 

          LS-PrePost

          LS-PrePost is an ad­vanced pre and post-proces­sor that is de­liv­ered free with LS-DY­NA.

          Availability and Restrictions

          Versions

          The following versions of ls-prepost are available on OSC clusters:

          Version Owens
          4.6 X*
          * Current default version

          You can use module spider ls-prepost to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          In order to use LS-PrePost you need LS-DYNA. OSC does not provide LS-DYNA license directly, however users with their own academic departmental license can use it on the OSC clusters.  Please contact OSC Help for further instruction.

          Publisher/Vendor/Repository and License Type

          LSTC, Commercial

          Usage

          Usage on Owens

          Set-up

          To configure your environment for use of LS-PrePost, run the following command: module load ls-prepost. The default version will be loaded. To select a particular LS-PrePost version, use module load ls-prepost/<version>. For example, use module load ls-prepost/4.6 to load LS-PrePost 4.6.

          Further Reading

          Supercomputer: 
          Service: 

          User-Defined Material for LS-DYNA

          This page describes how to specify user defined material to use within LS-DYNA.  The user-defined subroutines in LS-DYNA allow the program to be customized for particular applications.  In order to define user material, LS-DYNA must be recompiled.

          Usage

          The first step to running a simulation with user defined material is to build a new executable. The following is an example done with solver version mpp971_s_R7.1.1.

          When you log into the Oakley system, load mpp971_s_R7.1.1 with the command:

          module load mpp-dyna/R7.1.1

          Next, copy the mpp971_s_R7.1.1 object files and Makefile to your current directory:

          cp /usr/local/lstc/mpp-dyna/R7.1.1/usermat/* $PWD

          Next, update the dyn21.f file with your user defined material model subroutine. Please see the LS-DYNA User's Manual (Keyword version) for details regarding the format and structure of this file.

          Once your user defined model is setup correctly in dyn21.f, build the new mpp971 executable with the command:

          make

          To execute a multi processor (ppn > 1) run with your new executable, execute the following steps:

          1) move your input file to a directory on an OSC system (pipe.k in the example below)

          2) copy your newly created mpp971 executable to this directory as well

          3) create a batch script (lstc_umat.job) like the following:

          #PBS -N LSDYNA_umat
          #PBS -l walltime=1:00:00
          #PBS -l nodes=2:ppn=8
          #PBS -j oe
          #PBS -S /bin/csh
          
          # This is the template batch script for running a pre-compiled
          # MPP 971 v7600 LS-DYNA.  
          # Total number of processors is ( nodes x ppn )
          #
          # The following lines set up the LSDYNA environment
          module load mpp-dyna/R7.1.1
          #
          # Move to the directory where the job was submitted from
          # (i.e. PBS_O_WORKDIR = directory where you typed qsub)
          #
          cd $PBS_O_WORKDIR
          #
          # Run LSDYNA 
          # NOTE: you have to put in your input file name
          #
          mpiexec mpp971 I=pipe.k NCPU=16

                    4) Next, submit this job to the batch queue with the command:

                 qsub lstc_umat.job

          The output result files will be saved to the directory you ran the qsub command from (known as the $PBS_O_WORKDIR_

          Documentation

          On-line documentation is available on LSTC website.

          See Also

           

           

          Service: 

          MAGMA

          MAGMA is a collection of next generation linear algebra (LA) GPU accelerated libraries designed and implemented by the team that developed LAPACK and ScaLAPACK. MAGMA is for heterogeneous GPU-based architectures, it supports interfaces to current LA packages and standards, e.g., LAPACK and BLAS, to allow computational scientists to effortlessly port any LA-relying software components. The main benefits of using MAGMA are that it can enable applications to fully exploit the power of current heterogeneous systems of multi/manycore CPUs and multi-GPUs, and deliver the fastest possible time to an accurate solution within given energy constraints.

          Availability and Restrictions

          Versions

          MAGMA is available on Owens, and the following versions are currently available at OSC:

          Version Owens
          2.2.0 X(I)*
          * Current default version; I = avaible with only intel

          You can use module spider magma to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access for Academic Users

          MAGMA is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Computational Algebra Group, Univ. of Sydney, Open source

          Usage

          Usage on Owens

          Set-up

          To load the default version of MAGMA module, cuda must first be loaded. Use module load cuda toload the default version of cuda, or module load cuda/version to load a specific version. Then use module load magma to load MAGMA. To select a particular software version, use module load magma/version. For example, use   module load magma/2.2.0 to load MAGMA version 2.2.0

          Using MAGMA

          To run MAGMA in the command line, use the Intel compilers (icc, ifort). 

          icc $MAGMA_CFLAGS example.c
          

          or

          ifort $MAGMA_F90FLAGS example.F90
          

          Batch Usage

          When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your MAGMA simulation to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

          Interactive Batch Session
          For an interactive batch session, one can run the following command:
          qsub -I -l nodes=1:ppn=28 -l walltime=1:00:00
          
          which gives you 28 cores (-l nodes=1:ppn=28) with 1 hour (-l walltime=1:00:00). You may adjust the numbers per your need.
          Non-interactive Batch Job (Serial Run)

          batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice.

          Below is the example batch script (job.txt) for a serial run:

          # MAGMA Example Batch Script for the Basic Tutorial in the MAGMA manual
          #PBS -N 6pti
          #PBS -l nodes=1:ppn=28
          #PBS -l walltime=0:20:00
          module load cuda
          module load magma
          # Use TMPDIR for best performance.
          cd $TMPDIR
          # PBS_O_WORKDIR refers to the directory from which the job was submitted.
          cp $PBS_O_WORKDIR/example.c .
          icc $MAGMA_CFLAGS example.c
          

          In order to run it via the batch system, submit the job.txt file with the command: qsub job.txt

          Further Reading

          Tag: 
          Supercomputer: 
          Service: 
          Technologies: 
          Fields of Science: 

          MATLAB

          MATLAB is a technical computing environment for high-performance numeric computation and visualization. MATLAB integrates numerical analysis, matrix computation, signal processing, and graphics in an easy-to-use environment where problems and solutions are expressed just as they are written mathematically--without traditional programming.

          Availability and Restrictions

          Versions

          MATLAB is available on Pitzer and Owens Clusters. The versions currently available at OSC are:

          Version Owens Pitzer Notes
          2015b X    
          2016b X    
          2017a X    
          2018a X X  
          2018b X X  
          2019a   X  
          2019b X X  
          2020a X* X*  
          * Current default version

          You can use module spider matlab to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access: Academic Users Only (non-commercial, non-government)

          Any academic users at OSC can use Matlab. All users must be added to the license server before using MATLAB. Please contact OSC Help to be granted access.

          Publisher/Vendor/Repository and License Type

          MathWorks, Commercial (University site license)

          Usage

          Usage on Owens

          Set-up

          To load the default version of MATLAB module, use  module load matlab . For a list of all available MATLAB versions and the format expected, type:  module spider matlab . To select a particular software version, use   module load matlab/version . For example, use  module load matlab/r2015b  to load MATLAB version r2015b. 

          Running MATLAB

          The following command will start an interactive, command line version of MATLAB:

          matlab -nodisplay 
          
          If you are able to use X-11 forwarding and have enabled it in your SSH client software preferences, you can run MATLAB using the GUI by typing the command  matlab . For more information about the matlab command usage, type  matlab –h  for a complete list of command line options.

          The commands listed above will run MATLAB on the login node you are connected to. As the login node is a shared resource, running scripts that require significant computational resources will impact the usability of the cluster for others. As such, you should not use interactive MATLAB sessions on the login node for any significant computation. If your MATLAB script requires significant time, CPU power, or memory, you should run your code via the batch system.

          Batch Usage

          When you log into owens.osc.edu you are actually logged into a Linux box referred to as the login node. To gain access to the multiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request multiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

          Interactive Batch Session
          For an interactive batch session using the command line version of MATLAB, one can run the following command:
          sinteractive -A <project-account> -N 1 -n 28 -t 00:20:00
          

          which requests one whole node with 28 cores ( -N 1 -n 28 ), for a walltime of 20 minutes ( -t 00:20:00 ). Here you can run MATLAB interactively by loading the MATLAB module and running MATLAB with the options of your choice as described above. You may adjust the numbers per your need.

          Parallel Processing in MATLAB

          MATLAB supports implicit multithreading on a single node.  

          Multithreading

          Multithreading allows some functions in MATLAB to distribute the work load between cores of the node that your job is running on. By default, all of the current versions of MATLAB available on the OSC clusters have multithreading enabled. 

          The system will run as many threads as there are cores on the nodes requested. 

          Multithreading increases the speed of some linear algebra routines, but if you would like to disable multithreading you may include the option " -singleCompThread " when running MATLAB. An example is given below:

          #!/bin/bash
          #SBATCH --job-name disable_multithreading
          #SBATCH --time=00:10:00
          #SBATCH --nodes=1 --ntasks-per-node=28
          #SBATCH --account=<project-account>
          
          module load matlab
          matlab -singleCompThread -nodisplay -nodesktop < hello.m
          # end of example file
          

           

          Usage on Pitzer

          Set-up

          To load the default version of MATLAB module, use module load matlab.

          Running MATLAB

          The following command will start an interactive, command line version of MATLAB:

          matlab -nodisplay 
          
          If you are able to use X-11 forwarding and have enabled it in your SSH client software preferences, you can run MATLAB using the GUI by typing the command  matlab. For more information about the matlab command usage, type  matlab –h for a complete list of command line options.

          The commands listed above will run MATLAB on the login node you are connected to. As the login node is a shared resource, running scripts that require significant computational resources will impact the usability of the cluster for others. As such, you should not use interactive MATLAB sessions on the login node for any significant computation. If your MATLAB script requires significant time, CPU power, or memory, you should run your code via the batch system.

          Batch Usage

          When you log into pitzer.osc.edu you are actually logged into a Linux box referred to as the login node. To gain access to the multiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request multiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

          Interactive Batch Session
          For an interactive batch session using the command line version of MATLAB, one can run the following command:
          sinteractive -A <project-account> -N 1 -n 40 -t 00:20:00
          

          which requests one whole node with 40 cores ( -N 1 -n 40), for a walltime of 20 minutes ( -t 00:20:00 ). Here you can run MATLAB interactively by loading the MATLAB module and running MATLAB with the options of your choice as described above. You may adjust the numbers per your need.

          Parallel Processing in MATLAB

          MATLAB supports implicit multithreading on a single node.  

          Multithreading

          Multithreading allows some functions in MATLAB to distribute the work load between cores of the node that your job is running on. By default, all of the current versions of MATLAB available on the OSC clusters have multithreading enabled. 

          The system will run as many threads as there are cores on the nodes requested.

          Multithreading increases the speed of some linear algebra routines, but if you would like to disable multithreading you may include the option " -singleCompThread" when running MATLAB. An example is given below:

          #!/bin/bash
          #SBATCH --job-name disable_multithreading
          #SBATCH --time=00:10:00
          #SBATCH --nodes=1 --ntasks-per-node=40
          #SBATCH --account=<project-account>
          
          module load matlab
          matlab -singleCompThread -nodisplay -nodesktop < hello.m
          # end of example file
          

          MATLAB with a GPU

          A GPU can be utilized for MATLAB. You can acquire a GPU by

          #SBATCH --gpus-per-node=1

          for Owens, or Pitzer. For more detail, please read here.

          You can check the GPU assigned to you using:

          gpuDeviceCount  # show how many GPUs you have
          gpuDevice       # show the details of the GPU
          

          You can replace an array to gpuArray, for example:

          A = gpuArray(A)
          

          where A is a regular MATLAB array. This transfers the array data to the GPU memory. Then, you can use the gpuArray variable in GPU supported built-in functions. You can find the full list of GPU supported built-in functions from here. For more information about GPU programming for MATLAB, please read "GPU Computing" from Mathworks.

          Toolboxes and Features

          OSC's current licenses support the following MATLAB toolboxes and features (please contact OSC Help for license-specific questions):

          Polyspace Bug Finder
          PolySpace_Bug_Finder_Engine
          Aerospace_Blockset
          Aerospace_Toolbox
          Antenna_Toolbox
          Audio_System_Toolbox
          Automated_Driving_Toolbox
          Bioinformatics_Toolbox
          Communications System Toolbox
          Computer Vision System Toolbox
          Control System Toolbox
          Curve_Fitting_Toolbox
          DSP System Toolbox
          Data Acquisition Toolbox
          Database_Toolbox
          Datafeed_Toolbox
          Econometrics_Toolbox
          RTW_Embedded_Coder
          Filter_Design_HDL_Coder
          Financial Instruments Toolbox
          Financial_Toolbox
          Fixed_Point_Toolbox
          Fuzzy Logic Toolbox
          Global Optimization Toolbox
          Simulink_HDL_Coder
          EDA_Simulator_Link
          Image_Acquisition_Toolbox
          Image Processing Toolbox
          Instrument Control Toolbox
          LTE System Toolbox
          MATLAB_Coder
          MATLAB_Builder_for_Java
          Compiler
          MATLAB Report Generator
          Mapping Toolbox
          Model Predictive Control Toolbox
          Model-Based Calibration Toolbox
          Neural_Network_Toolbox
          OPC_Toolbox
          Optimization_Toolbox
          Distrib_Computing_Toolbox (Parallel Computing Toolbox)
          Partial Differential Equation Toolbox
          Phased_Array_System_Toolbox
          Polyspace Code Prover
          Powertrain_Blockset
          RF_Blockset
          RF_Toolbox
          Risk_Management_Toolbox
          Robotics_System_Toolbox
          Robust Control Toolbox
          Signal Processing Toolbox
          SimBiology
          SimEvents
          Simscape Driveline
          Simscape Electronics
          Simscape Fluids
          Simscape Multibody
          Simscape Power Systems
          Simscape
          Virtual_Reality_Toolbox
          Simulink_Code_Inspector
          Real-Time_Workshop
          Simulink_Control_Design
          Simulink Design Optimization
          Simulink_Design_Verifier
          Simulink Desktop Real-Time
          Simulink_PLC_Coder
          XPC_Target
          Simulink Report Generator
          Simulink_Test
          Simulink Verification and Validation
          Excel_Link
          Stateflow
          Statistics and Machine Learning Toolbox
          Symbolic Math Toolbox
          System Identification Toolbox
          Trading_Toolbox
          Vehicle_Network_Toolbox
          Vision_HDL_Toolbox
          WLAN_System_Toolbox
          Wavelet_Toolbox
          

          See this page if you need to install additional toolbox by yourself. 

          Further Reading

          Official PDF documentation can be obtained from the MathWorks Website

          Supercomputer: 
          Service: 
          Fields of Science: 

          MIRA

          MIRA - Sequence assembler and sequence mapping for whole genome shotgun and EST / RNASeq sequencing data. Can use Sanger, 454, Illumina and IonTorrent data. PacBio: CCS and error corrected data usable, uncorrected not yet.

          Availability and Restrictions

          Versions

          The following versions of MIRA are available on OSC clusters:

          Version Owens
          4.0.2 X*
          * Current default version

          You can use module spider mira to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          MIRA is available to all OSC users. If you have any questions, please contact OSC Help.

          Publisher/Vendor/Repository and License Type

          Bastien Chevreux, Open source

          Usage

          Usage on Owens

          Set-up

          To configure your environment for use of MIRA, run the following command: module load mira. The default version will be loaded. To select a particular MIRA version, use module load mira/version. For example, use module load mira/4.0.2 to load MIRA 4.0.2.

          Further Reading

          Supercomputer: 
          Service: 
          Fields of Science: 

          MKL - Intel Math Kernel Library

          Intel Math Kernel Library (MKL) consists of high-performance, multithreaded mathematics libraries for linear algebra, fast Fourier transforms, vector math, and more.

          Availability and Restrictions

          Versions

          OSC supports single-process use of MKL for LAPACK and BLAS levels one through three. For multi-process applications, we also support the ScaLAPACK, FFTW2, and FFTW3 MKL wrappers. MKL modules are available for the Intel, GNU, and PGI compilers. MKL is available on Pitzer, Ruby, and Owens Clusters. The versions currently available at OSC are:

          Version Owens Pitzer Notes
          11.3.2 X    
          11.3.3 X    
          2017.0.2 X    
          2017.0.4 X    
          2017.0.7   X  
          2018.0.3 X X  
          2019.0.3 X X  
          2019.0.5 X* X*  
          * Current Default Version

          You can use module spider mkl to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

          Access

          MKL is available to all OSC users.

          Publisher/Vendor/Repository and License Type

          Intel, Commercial

          Usage

          Usage on Owens

          Set-up

          To load the default MKL, run the following command: module load mkl. To load a particular version, use  module load mkl/version. For example, use module load mkl/11.3.3 to load MKL version 11.3.3. You can use module spider mkl to view available modules.

          This step is required for both building and running MKL applications. Note that loading an mkl module defines several environment variables that can be useful for compiling and linking to MKL, e.g., MKL_CFLAGS and MKL_LIBS.

          Exception: The "mkl" module is usually not needed when using the Intel compilers; just use the "-mkl" flag on the compile and link steps.

          Usage on Pitzer

          Set-up

          To load the default MKL, run the following command: module load mkl

          This step is required for both building and running MKL applications.  Note that loading an mkl module defines several environment variables that can be useful for compiling and linking to MKL, e.g., MKL_CFLAGS and MKL_LIBS.

          Exception: The "mkl" module is usually not needed when using the Intel compilers; just use the "-mkl" flag on the compile and link steps.

          Dynamic Linking Variables

          These variables indicate how to link to MKL.  While their contents are used during compiling and linking, the variables themselves are usually specified during the configuration stage of software installation.  The form of specification is dependent on the application software.  For example, some softwares employing cmake for configuration might use this form:

          cmake ..  -DMKL_INCLUDE_DIR="$MKLROOT/include"  -DMKL_LIBRARIES="MKL_LIBS_SEQ" 

          Here is an exmple for some software employing autoconf:

          ./configure --prefix=$HOME/local/pkg/version CPPFLAGS="$MKL_CFLAGS" LIBS="$MKL_LIBS" LDFLAGS="$MKL_LIBS"

           

          Variable Comment
          MKL_LIBS Link with parallel threading layer of MKL
          GNU_MKL_LIBS Dedicated for GNU compiler in Intel programming environment
          MKL_LIBS_SEQ Link with sequential threading layer of MKL
          MKL_SCALAPACK_LIBS Link with BLACS and ScaLAPACK of MKL
          MKL_CLUSTER_LIBS Link with BLACS, CDFT and ScaLAPACK of MK